Oct 30 02:41:18.820705 ip-10-0-133-197 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:41:19.170807 ip-10-0-133-197 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:19.170807 ip-10-0-133-197 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:41:19.170807 ip-10-0-133-197 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:19.170807 ip-10-0-133-197 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:41:19.174929 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174851    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:19.174929 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174928    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:19.174929 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174932    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174935    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174938    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174941    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174944    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174946    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174949    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174952    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174954    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174957    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174960    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174963    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174965    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174968    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174971    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174973    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174976    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174978    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174981    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174984    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:19.175020 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174986    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174990    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174992    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.174995    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175005    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175009    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175011    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175014    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175017    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175019    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175022    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175026    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175029    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175031    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175034    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175037    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175040    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175042    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175045    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:19.175499 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175047    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175052    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175056    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175059    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175062    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175065    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175068    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175071    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175074    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175076    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175080    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175082    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175085    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175088    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175091    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175094    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175097    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175100    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175103    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:19.175940 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175105    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175108    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175111    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175113    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175116    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175118    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175121    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175124    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175126    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175129    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175131    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175134    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175137    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175139    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175142    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175144    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175147    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175149    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175152    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175154    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:19.176392 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175157    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175159    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175162    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175164    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175167    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175171    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175176    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175179    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175182    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175185    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175188    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175190    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175194    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175197    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175199    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175202    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175205    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175208    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175211    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175213    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:19.176896 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175216    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175218    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175221    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175226    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175229    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.175231    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176611    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176620    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176623    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176627    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176630    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176632    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176636    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176640    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176644    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176646    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176649    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176651    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176654    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:19.177395 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176657    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176660    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176663    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176666    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176669    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176671    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176674    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176676    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176679    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176682    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176684    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176687    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176690    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176692    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176695    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176697    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176700    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176703    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176705    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176708    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:19.177895 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176710    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176713    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176715    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176718    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176720    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176723    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176725    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176728    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176730    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176733    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176735    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176738    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176740    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176743    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176747    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176750    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176752    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176755    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176757    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176760    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:19.178377 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176762    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176765    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176768    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176773    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176776    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176779    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176783    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176785    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176788    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176791    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176793    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176796    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176798    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176801    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176803    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176806    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176808    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176811    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176813    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:19.178872 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176816    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176818    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176821    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176823    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176826    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176828    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176831    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176834    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176837    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176839    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176842    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176844    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176847    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176850    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176853    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176855    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176858    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176860    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176863    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:19.179360 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176867    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176869    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176872    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176874    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176876    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176879    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176881    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176884    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176886    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176889    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176891    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176894    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176896    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176899    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.176902    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177578    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177587    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177590    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177594    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177597    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:19.179862 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177601    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177604    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177607    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177610    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177613    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177616    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177619    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177621    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177624    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177627    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177630    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177632    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177635    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177637    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177640    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177642    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177645    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177647    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177650    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177653    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:19.180334 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177655    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177658    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177660    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177663    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177666    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177668    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177671    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177673    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177676    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177678    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177682    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177684    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177687    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177690    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177693    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177696    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177699    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177701    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177704    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177706    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:19.180819 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177709    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177712    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177714    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177717    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177719    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177722    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177724    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177727    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177729    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177732    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177734    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177737    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177739    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177742    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177744    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177747    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177749    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177752    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177755    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:19.181295 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177757    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177760    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177762    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177765    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177768    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177772    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177776    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177780    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177783    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177786    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177789    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177791    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177794    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177796    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177799    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177801    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177804    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177806    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177809    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:19.181756 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177811    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177814    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177817    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177819    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177822    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177825    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177827    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177833    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177836    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177841    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177845    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177848    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177850    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177853    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177856    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177858    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177861    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177863    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177866    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:19.182221 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177869    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177871    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177874    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.177877    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178728    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178738    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178748    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178753    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178758    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178762    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178766    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178770    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178774    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178778    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178781    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178785    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178788    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178791    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178794    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178797    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178801    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178804    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178808    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:41:19.182693 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178811    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178814    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178817    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178821    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178825    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178828    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178831    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178834    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178837    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178840    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178844    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178847    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178850    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178855    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178858    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178861    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178864    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178867    2571 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178870    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178874    2571 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178878    2571 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178881    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178884    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178887    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:41:19.183267 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178892    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178895    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178898    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178901    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178904    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178909    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178913    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178916    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178919    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178922    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178925    2571 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178929    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178932    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178935    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178938    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178941    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178944    2571 flags.go:64] FLAG: --help="false"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178947    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178951    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178954    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178956    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178960    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178964    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178967    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:41:19.183895 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178970    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178973    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178976    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178979    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178982    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178985    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178988    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178990    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178993    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.178997    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179000    2571 flags.go:64] FLAG: --lock-file=""
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179003    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179006    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179010    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179016    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179019    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179022    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179025    2571 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179028    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179031    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179034    2571 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179037    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179041    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179044    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179049    2571 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:41:19.184492 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179052    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179055    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179058    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179061    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179064    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179067    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179070    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179077    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179080    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179083    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179086    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179089    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179092    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179095    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179098    2571 flags.go:64] FLAG: --port="10250"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179101    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179104    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-006b16d6b6dbf3084"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179107    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179110    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179113    2571 flags.go:64] FLAG: --register-node="true"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179118    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179121    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179124    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179127    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179130    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179134    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:41:19.185164 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179137    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179140    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179143    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179146    2571 flags.go:64] FLAG: --runonce="false"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179149    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179152    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179155    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179158    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179161    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179164    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179168    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179171    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179174    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179177    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179180    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179183    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179186    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179189    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179192    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179197    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179200    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179203    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179208    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179211    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179214    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:41:19.185804 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179217    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179222    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179225    2571 flags.go:64] FLAG: --v="2"
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179229    2571 flags.go:64] FLAG: --version="false"
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179233    2571 flags.go:64] FLAG: --vmodule=""
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179237    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179240    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179379    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179383    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179387    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179389    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179392    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179395    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179398    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179401    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179403    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179406    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179409    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179411    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179414    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179417    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:19.186388 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179420    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179440    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179442    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179445    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179447    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179450    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179453    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179455    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179459    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179462    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179465    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179467    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179471    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179474    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179477    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179479    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179482    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179485    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179489    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179493    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:19.186909 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179496    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179499    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179502    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179504    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179507    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179510    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179512    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179515    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179517    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179520    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179523    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179525    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179528    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179531    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179534    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179537    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179540    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179543    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179545    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179548    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:19.187396 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179552    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179554    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179557    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179559    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179563    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179567    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179571    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179574    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179578    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179580    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179583    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179587    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179590    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179592    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179595    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179598    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179600    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179603    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179605    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179608    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:19.187881 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179611    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179613    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179616    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179619    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179621    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179624    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179627    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179629    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179632    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179634    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179637    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179640    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179644    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179646    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179650    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179652    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179656    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179659    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179661    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:19.188391 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179664    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179667    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179669    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179672    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179674    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179677    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179679    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179682    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179684    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179687    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179690    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179692    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179695    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.179721    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179771    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179776    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:19.188864 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179779    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179782    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179785    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179788    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179791    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179794    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179797    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179799    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179802    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179804    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179809    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179811    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179814    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179818    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179820    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179823    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179826    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179828    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179831    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:19.189245 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179833    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179836    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179839    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179842    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179844    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179847    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179850    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179852    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179855    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179857    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179860    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179863    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179865    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179868    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179871    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179874    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179876    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179879    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179882    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:19.189708 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179885    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179887    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179890    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179892    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179897    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179901    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179905    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179908    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179911    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179914    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179917    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179919    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179922    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179925    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179927    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179930    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179933    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179935    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179938    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:19.190172 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179941    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179943    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179946    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179949    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179951    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179954    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179957    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179959    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179962    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179964    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179967    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179970    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179975    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179979    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179982    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179986    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179989    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179993    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179996    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.179998    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:19.190654 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180001    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180004    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180007    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180009    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180012    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180014    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180017    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180019    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180022    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180025    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180028    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180030    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180033    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180035    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180038    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180040    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180043    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180046    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180049    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180051    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:19.191130 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180054    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180057    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180059    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180062    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180065    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180067    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.180070    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.186369    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.186384    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.187025    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.187031    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187088    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187094    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187097    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187101    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187104    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187106    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187109    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187112    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187115    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:19.191780 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187118    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187121    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187123    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187126    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187129    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187132    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187134    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187141    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187144    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187146    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187149    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187152    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187154    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187158    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187161    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187164    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187167    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187170    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187173    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:19.192365 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187176    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187179    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187181    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187184    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187187    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187190    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187193    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187196    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187198    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187201    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187203    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187207    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187212    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187215    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187218    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187221    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187224    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187226    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187229    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:19.192897 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187231    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187234    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187237    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187240    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187242    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187245    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187248    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187251    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187253    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187256    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187259    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187261    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187264    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187267    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187272    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187276    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187280    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187283    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187286    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:19.193373 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187289    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187291    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187294    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187297    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187299    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187302    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187305    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187307    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187310    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187312    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187315    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187318    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187320    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187323    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187326    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187329    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187332    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187335    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187337    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187340    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:19.193829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187342    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187345    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187347    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187350    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187353    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187355    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187358    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187361    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187364    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187366    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187369    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187372    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187374    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187377    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187379    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187382    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187384    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187387    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187389    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:19.194381 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187392    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.187420    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187570    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187576    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187581    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187585    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187588    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187591    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187594    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187597    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187600    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187602    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187606    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187611    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187615    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:19.194886 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187618    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187621    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187624    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187626    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187629    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187632    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187635    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187637    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187640    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187642    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187645    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187648    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187650    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187653    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187655    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187658    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187660    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187663    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187665    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:19.195248 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187668    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187670    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187673    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187676    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187679    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187681    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187684    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187686    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187689    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187691    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187694    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187697    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187699    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187702    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187704    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187707    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187709    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187712    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187714    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187718    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:19.195836 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187721    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187723    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187726    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187728    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187731    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187734    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187736    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187739    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187741    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187744    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187746    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187749    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187751    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187754    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187756    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187759    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187763    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187766    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187769    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:19.196314 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187772    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187774    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187777    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187780    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187782    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187785    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187787    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187790    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187792    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187795    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187797    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187800    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187802    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187805    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187808    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187810    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187813    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187815    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187818    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187820    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:19.196760 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187823    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187825    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187828    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187830    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187833    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187835    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187838    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187841    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187844    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187846    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187849    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187852    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187854    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187857    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187859    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.187882    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:41:19.197354 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187915    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187919    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187922    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187925    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187928    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187931    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187934    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187936    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187939    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187942    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187945    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187947    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187951    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187955    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187958    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187961    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187963    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187966    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187969    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:41:19.197803 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187971    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187974    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187977    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187979    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187982    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187985    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187987    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187990    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187993    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187996    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.187998    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188001    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188003    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188006    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188009    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188012    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188014    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188016    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188019    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188022    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:41:19.198244 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188024    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188026    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188029    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188031    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188035    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188037    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188040    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188042    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188045    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188047    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188050    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188052    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188055    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188058    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188060    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188066    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188069    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188071    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188074    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:41:19.198738 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188076    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188079    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188083    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188086    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188090    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188093    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188095    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188098    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188100    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188103    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188105    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188108    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188110    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188112    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188115    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188117    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188120    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188122    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188125    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:41:19.199370 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188128    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188130    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188133    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188136    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188138    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188140    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188143    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188146    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188149    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188153    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188156    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188158    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188161    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188164    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188166    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188169    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188172    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188175    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188177    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188180    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:41:19.199829 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188182    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188184    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188187    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188190    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188192    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188195    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188199    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188202    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:19.188204    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.188252    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.190187    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.191016    2571 server.go:1014] "Starting client certificate rotation"
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.191103    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.191619    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.194634    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:19.200298 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.197454    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:41:19.206805 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.206788    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:41:19.209587 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.209573    2571 log.go:25] "Validated CRI v1 image API"
Oct 30 02:41:19.210497 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.210396    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:41:19.212936 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.212906    2571 fs.go:135] Filesystem UUIDs: map[247bbc2a-230c-4710-ab71-ee258ccd3310:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ce097412-dd54-46ee-b68b-56b240f00369:/dev/nvme0n1p3]
Oct 30 02:41:19.212936 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.212929    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:41:19.218712 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.218609    2571 manager.go:217] Machine: {Timestamp:2026-10-30 02:41:19.217329284 +0000 UTC m=+0.290012774 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096973 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9636410522fa34c0a2cf3ad88943 SystemUUID:ec2a9636-4105-22fa-34c0-a2cf3ad88943 BootID:5e824f4c-6520-45b2-963d-8ba7d110686f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:b9:a8:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:b9:a8:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:88:a5:df:32:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:41:19.218712 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.218709    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:41:19.218821 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.218777    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:41:19.221269 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.221248    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:41:19.221389 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.221270    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:41:19.221454 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.221397    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:41:19.221454 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.221405    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:41:19.221454 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.221445    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:19.221527 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.221457    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:41:19.222348 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.222337    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:41:19.226252 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.226240    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:41:19.226370 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.226361    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:41:19.226993 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.226982    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:41:19.227023 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.226999    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:41:19.227023 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.227009    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:41:19.227023 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.227018    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:41:19.227123 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.227026    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:41:19.227411 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.227392    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:19.230832 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.230782    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:41:19.235612 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.235595    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:41:19.236521 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.236508    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:41:19.236579 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.236539    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:41:19.236700 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.236690    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:41:19.236732 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.236706    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:41:19.236732 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.236712    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:41:19.236732 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.236718    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:41:19.236732 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.236723    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:41:19.236732 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.236730    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:41:19.237160 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.237152    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:41:19.237192 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.237161    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:41:19.237192 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.237168    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:41:19.237192 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.237173    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:41:19.237192 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.237182    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:41:19.237192 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.237191    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:41:19.238525 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.238508    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:41:19.238525 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.238526    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:41:19.242472 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.242450    2571 server.go:1263] "Started kubelet"
Oct 30 02:41:19.242571 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.242534    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:41:19.242642 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.242532    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:41:19.242692 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.242641    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:41:19.243188 ip-10-0-133-197 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:41:19.244173 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.243815    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:41:19.244263 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.244252    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:41:19.246615 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.246582    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:41:19.249596 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.249579    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:41:19.249996 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.249980    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:41:19.250078 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.250062    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:19.250522 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.250501    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:41:19.250522 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.250521    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:41:19.250657 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.250553    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:41:19.250657 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.250622    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:41:19.250657 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.250631    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:41:19.250868 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.250850    2571 factory.go:55] Registering systemd factory
Oct 30 02:41:19.250921 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.250870    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:41:19.251089 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.251074    2571 factory.go:153] Registering CRI-O factory
Oct 30 02:41:19.251089 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.251091    2571 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:41:19.251193 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.251149    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:41:19.251193 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.251164    2571 factory.go:103] Registering Raw factory
Oct 30 02:41:19.251193 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.251181    2571 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:41:19.252000 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.251986    2571 manager.go:319] Starting recovery of all containers
Oct 30 02:41:19.262656 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.262637    2571 manager.go:324] Recovery completed
Oct 30 02:41:19.267704 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.267690    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.269190 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.269168    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.269270 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.269203    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.269270 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.269219    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.269613 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.269602    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:41:19.269613 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.269612    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:41:19.269726 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.269628    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:41:19.269726 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.269671    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.270923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.270905    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.270989 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.270932    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.270989 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.270945    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.271799 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.271786    2571 policy_none.go:50] "Start"
Oct 30 02:41:19.271847 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.271802    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:41:19.271847 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.271816    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:41:19.273813 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.273801    2571 policy_none.go:44] "Start"
Oct 30 02:41:19.281570 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.281547    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:41:19.281698 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.281525    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-197.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:41:19.295059 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.294181    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-197.ec2.internal.18e32e62f8fa1fd2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-197.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-197.ec2.internal,},FirstTimestamp:2026-10-30 02:41:19.242403794 +0000 UTC m=+0.315087284,LastTimestamp:2026-10-30 02:41:19.242403794 +0000 UTC m=+0.315087284,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-197.ec2.internal,}"
Oct 30 02:41:19.298345 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.298329    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:41:19.301065 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.301036    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:41:19.301726 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.301699    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:41:19.306344 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.306331    2571 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:41:19.306884 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.306865    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:41:19.306935 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.306886    2571 server.go:87] "Starting device plugin registration server"
Oct 30 02:41:19.307160 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.307148    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:41:19.307192 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.307164    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:41:19.307292 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.307279    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:41:19.307396 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.307349    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:41:19.307396 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.307359    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:41:19.307810 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.307780    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:41:19.307883 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.307840    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:19.318300 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.318282    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qfhx2"
Oct 30 02:41:19.323614 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.323550    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-197.ec2.internal.18e32e62fa92d222  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-197.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-197.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-197.ec2.internal,},FirstTimestamp:2026-10-30 02:41:19.26918813 +0000 UTC m=+0.341871621,LastTimestamp:2026-10-30 02:41:19.26918813 +0000 UTC m=+0.341871621,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-197.ec2.internal,}"
Oct 30 02:41:19.323714 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.323598    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 02:41:19.337317 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.337300    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qfhx2"
Oct 30 02:41:19.361770 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.341946    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-197.ec2.internal.18e32e62fa932cc8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-197.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-197.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-197.ec2.internal,},FirstTimestamp:2026-10-30 02:41:19.269211336 +0000 UTC m=+0.341894831,LastTimestamp:2026-10-30 02:41:19.269211336 +0000 UTC m=+0.341894831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-197.ec2.internal,}"
Oct 30 02:41:19.361770 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.342313    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:41:19.386354 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.386325    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:41:19.387512 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.387495    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:41:19.387592 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.387523    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:41:19.387592 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.387541    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:41:19.387592 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.387579    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:41:19.401329 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.401314    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:19.407459 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.407442    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.408196 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.408179    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.408263 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.408213    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.408263 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.408225    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.408263 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.408249    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.423765 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.423727    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.423765 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.423747    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-197.ec2.internal\": node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:19.481257 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.481239    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:19.488389 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.488373    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"]
Oct 30 02:41:19.488450 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.488440    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.489177 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.489162    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.489242 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.489193    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.489242 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.489205    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.491224 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491211    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.491353 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491341    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.491389 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491370    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.491849 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491835    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.491910 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491851    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.491910 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491868    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.491910 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491854    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.491910 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491881    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.491910 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.491889    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.493971 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.493954    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.494047 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.493984    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:41:19.494598 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.494584    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:41:19.494659 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.494609    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:41:19.494659 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.494622    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:41:19.527637 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.527622    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-197.ec2.internal\" not found" node="ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.530306 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.530289    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-197.ec2.internal\" not found" node="ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.551797 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.551770    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.551861 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.551804    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf118574a1cbe5f944e98682e2ff56b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.ec2.internal\" (UID: \"adf118574a1cbe5f944e98682e2ff56b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.551861 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.551821    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.582161 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.582144    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:19.652239 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.652220    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.652299 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.652252    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.652337 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.652301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.652337 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.652316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf118574a1cbe5f944e98682e2ff56b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.ec2.internal\" (UID: \"adf118574a1cbe5f944e98682e2ff56b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.652399 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.652358    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf118574a1cbe5f944e98682e2ff56b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.ec2.internal\" (UID: \"adf118574a1cbe5f944e98682e2ff56b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.652399 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.652392    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.682261 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.682214    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:19.782414 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.782393    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:19.830708 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.830692    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.834019 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:19.834003    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:19.883401 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.883260    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:19.983865 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:19.983812    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:20.084244 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:20.084215    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:20.184782 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:20.184752    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:20.190919 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:20.190899    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:41:20.191041 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:20.191024    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:41:21.299427 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:21.299370    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:41:21.359595 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.359562    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:36:20 +0000 UTC" deadline="2028-03-24 16:48:07.999790469 +0000 UTC"
Oct 30 02:41:21.359595 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.359594    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12278h6m46.640200081s"
Oct 30 02:41:21.370196 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:21.370168    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf118574a1cbe5f944e98682e2ff56b.slice/crio-d4a0e5035706b392b895bba8ba3b3974d1a35eb8ac87cfd474be90a847b46ff5 WatchSource:0}: Error finding container d4a0e5035706b392b895bba8ba3b3974d1a35eb8ac87cfd474be90a847b46ff5: Status 404 returned error can't find the container with id d4a0e5035706b392b895bba8ba3b3974d1a35eb8ac87cfd474be90a847b46ff5
Oct 30 02:41:21.370674 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:21.370654    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68cf21ce8d2089c87349d6b232245b0.slice/crio-b85b0e8c0d01624df34a81d113234c2a2e7e159659a33b7bb740ae92ace5148d WatchSource:0}: Error finding container b85b0e8c0d01624df34a81d113234c2a2e7e159659a33b7bb740ae92ace5148d: Status 404 returned error can't find the container with id b85b0e8c0d01624df34a81d113234c2a2e7e159659a33b7bb740ae92ace5148d
Oct 30 02:41:21.374709 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.374693    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:41:21.391816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.391796    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:21.404421 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.404373    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal" event={"ID":"adf118574a1cbe5f944e98682e2ff56b","Type":"ContainerStarted","Data":"d4a0e5035706b392b895bba8ba3b3974d1a35eb8ac87cfd474be90a847b46ff5"}
Oct 30 02:41:21.405410 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.405393    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal" event={"ID":"a68cf21ce8d2089c87349d6b232245b0","Type":"ContainerStarted","Data":"b85b0e8c0d01624df34a81d113234c2a2e7e159659a33b7bb740ae92ace5148d"}
Oct 30 02:41:21.464683 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.464662    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:21.498263 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.498247    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:21.499839 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.499826    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:41:21.519009 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.518991    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:41:21.543120 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.543104    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:21.625075 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:21.625025    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:22.243340 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.243304    2571 apiserver.go:52] "Watching apiserver"
Oct 30 02:41:22.265411 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.265381    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:41:22.277263 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.277237    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:41:22.277601 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.277582    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-zdxqj","kube-system/global-pull-secret-syncer-6xh6w","kube-system/konnectivity-agent-mkxt8","openshift-image-registry/node-ca-qb86g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vj6lt","openshift-dns/node-resolver-k5blc","openshift-multus/multus-pvqdb","kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal","openshift-multus/network-metrics-daemon-gztvh","openshift-network-diagnostics/network-check-target-v7c87","openshift-network-operator/iptables-alerter-8l6sk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg","openshift-multus/multus-additional-cni-plugins-hp7gt"]
Oct 30 02:41:22.280234 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.280208    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.282673 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.282651    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:22.283203 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.283177    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:22.284637 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.284612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:22.287009 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.286990    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.289606 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.289585    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.293054 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.292700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.294671 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.294650    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2wtbz\""
Oct 30 02:41:22.294954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.294936    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.296469 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.296027    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:41:22.296469 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.296259    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:41:22.298079 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.298058    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.299635 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.299588    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-stwqd\""
Oct 30 02:41:22.300537 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.300515    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.300626 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.300577    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:22.304222 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.303795    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:22.304222 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.303845    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:22.306391 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.305862    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.308274 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.308254    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.331119 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.330957    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:41:22.331119 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.331022    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6j9f"
Oct 30 02:41:22.343286 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.343264    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6j9f"
Oct 30 02:41:22.365841 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.365819    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:41:22.374125 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374101    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-var-lib-kubelet\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.374218 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-k8s-cni-cncf-io\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.374218 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374167    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-netns\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374319 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-tuned\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.374319 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374294    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.374402 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374345    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-multus\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.374402 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjn9\" (UniqueName: \"kubernetes.io/projected/c27c447f-f1fa-4d0e-809e-f2945b673c32-kube-api-access-9tjn9\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.374496 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-log-socket\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374496 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374460    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-bin\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374496 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-env-overrides\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374603 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374513    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jx4h\" (UniqueName: \"kubernetes.io/projected/cea2f02d-9272-44dc-acd9-d899a2d711a9-kube-api-access-6jx4h\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374603 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-host\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.374660 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374602    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.374660 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374627    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-var-lib-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374660 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-node-log\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374766 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovn-node-metrics-cert\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374766 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.374766 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysconfig\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.374848 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374802    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-sys\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.374848 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-bin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.374918 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374856    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-ovn\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.374918 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/172aafae-6ad9-420e-a213-a8bc8f2439fe-konnectivity-ca\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:22.374979 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:22.374979 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-kubelet\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.374979 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374966    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-sys-fs\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.375099 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.374993    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.375099 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375024    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.375099 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-host\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.375099 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-systemd-units\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.375210 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375097    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.375210 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375125    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-cni-binary-copy\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.375266 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375220    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-hostroot\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.375266 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375245    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-config\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.375321 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375270    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-system-cni-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.375321 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375297    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lgtw\" (UniqueName: \"kubernetes.io/projected/d90d1fba-6d54-49b3-8792-36688f2242d3-kube-api-access-2lgtw\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.375381 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-serviceca\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.375381 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375347    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-system-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.375381 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375373    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-script-lib\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.375498 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-conf-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.375498 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-multus-certs\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.375498 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375433    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-kubelet\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.375498 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375460    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818913a6-9255-4e66-b325-5ced6778ad64-hosts-file\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.375498 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-systemd\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375519    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-tmp\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375563    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-kubernetes\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375588    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-conf\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375644    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmgqm\" (UniqueName: \"kubernetes.io/projected/fb013b42-f40c-4df9-bd10-ccb8975e54dc-kube-api-access-jmgqm\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375677    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-systemd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-lib-modules\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375732    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-device-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.375816 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375783    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-host-slash\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-netns\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-registration-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375916    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-slash\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375970    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.375997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-kube-api-access-xnlwq\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msnzw\" (UniqueName: \"kubernetes.io/projected/05e47302-4160-472b-988d-60585665c3bf-kube-api-access-msnzw\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376052    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-cnibin\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376078    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-iptables-alerter-script\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x24p\" (UniqueName: \"kubernetes.io/projected/818913a6-9255-4e66-b325-5ced6778ad64-kube-api-access-9x24p\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-etc-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376225    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mpjv\" (UniqueName: \"kubernetes.io/projected/cf492558-e351-45c6-8772-2324e53b4a97-kube-api-access-9mpjv\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:22.376257 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376253    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818913a6-9255-4e66-b325-5ced6778ad64-tmp-dir\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376307    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-multus-daemon-config\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376333    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-socket-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/172aafae-6ad9-420e-a213-a8bc8f2439fe-agent-certs\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376387    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-modprobe-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376440    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-run\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-cnibin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376512    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-socket-dir-parent\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376552    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-etc-kubernetes\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376618    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-netd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-kubelet-config\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376669    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-dbus\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.376929 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.376701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhddf\" (UniqueName: \"kubernetes.io/projected/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-kube-api-access-jhddf\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.396135 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.396112    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:41:22.396245 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.396198    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:41:22.396504 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.396485    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:22.396621 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.396576    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:41:22.396687 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.396626    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:41:22.396740 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.396712    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:41:22.396887 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.396872    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:41:22.396972 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.396947    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hjgzr\""
Oct 30 02:41:22.397305 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.397286    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:22.397425 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.397411    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:41:22.397576 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.397564    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:41:22.400499 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.400479    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:22.400595 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.400483    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:41:22.400668 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.400626    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:41:22.400668 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.400638    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:41:22.400753 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.400689    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:41:22.400753 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.400695    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:41:22.400753 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.400690    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:41:22.400753 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.400740    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:41:22.406247 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.406231    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:41:22.406331 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.406304    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:41:22.407180 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407161    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:41:22.407317 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407303    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zz8vj\""
Oct 30 02:41:22.407387 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407352    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:41:22.407439 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407408    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kxfcp\""
Oct 30 02:41:22.407511 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407490    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-29rqz\""
Oct 30 02:41:22.407583 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407567    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:41:22.407630 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407622    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:41:22.407713 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407697    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zm8vf\""
Oct 30 02:41:22.407801 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407783    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:41:22.407868 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407856    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6bd55\""
Oct 30 02:41:22.407925 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.407912    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vzqmp\""
Oct 30 02:41:22.477664 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.477813 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.477813 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477695    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-host\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.477813 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477779    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-systemd-units\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.477813 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477828    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-cni-binary-copy\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477860    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-host\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-hostroot\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-config\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477895    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477912    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-system-cni-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lgtw\" (UniqueName: \"kubernetes.io/projected/d90d1fba-6d54-49b3-8792-36688f2242d3-kube-api-access-2lgtw\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-serviceca\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477977    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-system-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.478012 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478012    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-script-lib\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478053    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-conf-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478042    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478102    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-multus-certs\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478105    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-conf-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-kubelet\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818913a6-9255-4e66-b325-5ced6778ad64-hosts-file\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478177    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-systemd\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-tmp\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478350    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-kubernetes\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478359    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-conf\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmgqm\" (UniqueName: \"kubernetes.io/projected/fb013b42-f40c-4df9-bd10-ccb8975e54dc-kube-api-access-jmgqm\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478437    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-systemd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.478505 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478482    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-lib-modules\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-device-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478580    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-host-slash\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-netns\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478695    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-registration-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478735    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-slash\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478749    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-system-cni-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478761    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478829    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-kube-api-access-xnlwq\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msnzw\" (UniqueName: \"kubernetes.io/projected/05e47302-4160-472b-988d-60585665c3bf-kube-api-access-msnzw\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.477953    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-systemd-units\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.478984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-cnibin\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479021    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-iptables-alerter-script\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479030    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-config\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.479188 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479045    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x24p\" (UniqueName: \"kubernetes.io/projected/818913a6-9255-4e66-b325-5ced6778ad64-kube-api-access-9x24p\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479093    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-etc-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpjv\" (UniqueName: \"kubernetes.io/projected/cf492558-e351-45c6-8772-2324e53b4a97-kube-api-access-9mpjv\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818913a6-9255-4e66-b325-5ced6778ad64-tmp-dir\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479151    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-system-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479182    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479214    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-multus-daemon-config\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-socket-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479273    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/172aafae-6ad9-420e-a213-a8bc8f2439fe-agent-certs\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479297    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-modprobe-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-run\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479368    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-cnibin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-socket-dir-parent\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479424    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-serviceca\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479441    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-cni-binary-copy\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-etc-kubernetes\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479466    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818913a6-9255-4e66-b325-5ced6778ad64-tmp-dir\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.479954 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479509    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-hostroot\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479511    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-host-slash\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479530    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-etc-kubernetes\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479574    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-slash\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479578    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-multus-certs\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479616    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818913a6-9255-4e66-b325-5ced6778ad64-hosts-file\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.479622    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479628    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-systemd\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479739    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-run\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479784    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-cnibin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479806    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-device-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479826    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-registration-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479874    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-kubernetes\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.479992    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-conf\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480124    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-lib-modules\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480279    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-systemd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480334    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-cnibin\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480334    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-etc-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.480682 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480372    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480396    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-socket-dir-parent\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480496    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-script-lib\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480500    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480498    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-socket-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.480531    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret podName:033b7d72-7e35-47ee-9cf9-d9993d21f4be nodeName:}" failed. No retries permitted until 2026-10-30 02:41:22.980489365 +0000 UTC m=+3.039149895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret") pod "global-pull-secret-syncer-6xh6w" (UID: "033b7d72-7e35-47ee-9cf9-d9993d21f4be") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480561    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-netns\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480659    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-netd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480705    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-netd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480750    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-kubelet-config\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480787    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-dbus\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480870    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-multus-daemon-config\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480875    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-modprobe-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480922    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-kubelet-config\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.481577 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480935    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-kubelet\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.480981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhddf\" (UniqueName: \"kubernetes.io/projected/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-kube-api-access-jhddf\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481006    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-dbus\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481007    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481028    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-var-lib-kubelet\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481125    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-k8s-cni-cncf-io\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481128    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-iptables-alerter-script\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481138    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-var-lib-kubelet\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481169    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-netns\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481184    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-k8s-cni-cncf-io\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481211    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481224    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-tuned\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481256    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481271    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-netns\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481280    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-multus\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481307    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjn9\" (UniqueName: \"kubernetes.io/projected/c27c447f-f1fa-4d0e-809e-f2945b673c32-kube-api-access-9tjn9\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-log-socket\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.482335 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481338    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-bin\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481341    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-multus\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481372    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-env-overrides\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481406    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-log-socket\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481413    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-bin\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jx4h\" (UniqueName: \"kubernetes.io/projected/cea2f02d-9272-44dc-acd9-d899a2d711a9-kube-api-access-6jx4h\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481467    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-host\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481490    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-var-lib-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481535    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-node-log\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481561    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovn-node-metrics-cert\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481618    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-var-lib-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481612    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysconfig\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-sys\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481671    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-node-log\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481679    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-bin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.483115 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481705    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-ovn\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481726    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481765    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-sys\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481774    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-env-overrides\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481799    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-bin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481832    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-ovn\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.481728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/172aafae-6ad9-420e-a213-a8bc8f2439fe-konnectivity-ca\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.483314    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.483354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-kubelet\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.483382    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-sys-fs\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.483513    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-sys-fs\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.482732    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.483622    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.483674    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:22.98365897 +0000 UTC m=+3.042319480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.483260    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-host\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.483735    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-kubelet\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.484132    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/172aafae-6ad9-420e-a213-a8bc8f2439fe-konnectivity-ca\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:22.484923 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.484366    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysconfig\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.485743 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.485208    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovn-node-metrics-cert\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.485743 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.485600    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-tuned\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.487035 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.487009    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-tmp\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.487647 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.487621    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/172aafae-6ad9-420e-a213-a8bc8f2439fe-agent-certs\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:22.507420 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.507366    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x24p\" (UniqueName: \"kubernetes.io/projected/818913a6-9255-4e66-b325-5ced6778ad64-kube-api-access-9x24p\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.511583 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.511534    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-kube-api-access-xnlwq\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.511900 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.511877    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjn9\" (UniqueName: \"kubernetes.io/projected/c27c447f-f1fa-4d0e-809e-f2945b673c32-kube-api-access-9tjn9\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.512243 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.512222    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:22.512243 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.512246    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:22.512405 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.512261    2571 projected.go:196] Error preparing data for projected volume kube-api-access-f276b for pod openshift-network-diagnostics/network-check-target-v7c87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:22.512405 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.512290    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msnzw\" (UniqueName: \"kubernetes.io/projected/05e47302-4160-472b-988d-60585665c3bf-kube-api-access-msnzw\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.512405 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.512308    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jx4h\" (UniqueName: \"kubernetes.io/projected/cea2f02d-9272-44dc-acd9-d899a2d711a9-kube-api-access-6jx4h\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.512405 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.512316    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b podName:41c6efff-036c-44b7-b0ff-700a1503ec82 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:23.012301059 +0000 UTC m=+3.070961593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f276b" (UniqueName: "kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b") pod "network-check-target-v7c87" (UID: "41c6efff-036c-44b7-b0ff-700a1503ec82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:22.513738 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.513713    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhddf\" (UniqueName: \"kubernetes.io/projected/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-kube-api-access-jhddf\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.514814 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.514771    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lgtw\" (UniqueName: \"kubernetes.io/projected/d90d1fba-6d54-49b3-8792-36688f2242d3-kube-api-access-2lgtw\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.514933 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.514912    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmgqm\" (UniqueName: \"kubernetes.io/projected/fb013b42-f40c-4df9-bd10-ccb8975e54dc-kube-api-access-jmgqm\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.515095 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.515069    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpjv\" (UniqueName: \"kubernetes.io/projected/cf492558-e351-45c6-8772-2324e53b4a97-kube-api-access-9mpjv\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:22.590378 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.590351    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:41:22.594784 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.594555    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:41:22.601619 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.601593    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:22.603679 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:22.603659    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818913a6_9255_4e66_b325_5ced6778ad64.slice/crio-46158511a4fd6b69bd5d6a1d126e3e938f9a98c1407c0cef949f17a93a53b875 WatchSource:0}: Error finding container 46158511a4fd6b69bd5d6a1d126e3e938f9a98c1407c0cef949f17a93a53b875: Status 404 returned error can't find the container with id 46158511a4fd6b69bd5d6a1d126e3e938f9a98c1407c0cef949f17a93a53b875
Oct 30 02:41:22.611992 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.611915    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:41:22.614727 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:22.614703    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172aafae_6ad9_420e_a213_a8bc8f2439fe.slice/crio-b1d5c40431ac91b661df11beae61a7443d367df013cacb8dd5ce03f31796bd85 WatchSource:0}: Error finding container b1d5c40431ac91b661df11beae61a7443d367df013cacb8dd5ce03f31796bd85: Status 404 returned error can't find the container with id b1d5c40431ac91b661df11beae61a7443d367df013cacb8dd5ce03f31796bd85
Oct 30 02:41:22.636104 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.636073    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:41:22.638180 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.638158    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:41:22.639536 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.639520    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvqdb"
Oct 30 02:41:22.648666 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:22.648638    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb013b42_f40c_4df9_bd10_ccb8975e54dc.slice/crio-4b5081c1421016df115012cff35da83e865949a3c9eed804eb2149d6c1f3159f WatchSource:0}: Error finding container 4b5081c1421016df115012cff35da83e865949a3c9eed804eb2149d6c1f3159f: Status 404 returned error can't find the container with id 4b5081c1421016df115012cff35da83e865949a3c9eed804eb2149d6c1f3159f
Oct 30 02:41:22.651476 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.651299    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:41:22.653076 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:22.653052    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e47302_4160_472b_988d_60585665c3bf.slice/crio-e62347cc65e9895ee37abee92e3e2c5ee559b1614ddce93bf56aa7c64255430d WatchSource:0}: Error finding container e62347cc65e9895ee37abee92e3e2c5ee559b1614ddce93bf56aa7c64255430d: Status 404 returned error can't find the container with id e62347cc65e9895ee37abee92e3e2c5ee559b1614ddce93bf56aa7c64255430d
Oct 30 02:41:22.653639 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:22.653616    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90d1fba_6d54_49b3_8792_36688f2242d3.slice/crio-392583615beca64131500f95cba5389c07e5ae636fb3e9e928c40e1fac6e4b29 WatchSource:0}: Error finding container 392583615beca64131500f95cba5389c07e5ae636fb3e9e928c40e1fac6e4b29: Status 404 returned error can't find the container with id 392583615beca64131500f95cba5389c07e5ae636fb3e9e928c40e1fac6e4b29
Oct 30 02:41:22.661506 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:22.661474    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27c447f_f1fa_4d0e_809e_f2945b673c32.slice/crio-0e9a5297aafc8d2b81006c842e17021e5f5bbe3490566423696e9b7d5631916c WatchSource:0}: Error finding container 0e9a5297aafc8d2b81006c842e17021e5f5bbe3490566423696e9b7d5631916c: Status 404 returned error can't find the container with id 0e9a5297aafc8d2b81006c842e17021e5f5bbe3490566423696e9b7d5631916c
Oct 30 02:41:22.667183 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.667161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:41:22.670590 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.670573    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:22.680208 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:22.680184    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d43c51_c7ff_4751_9c3b_0eeae8d6a355.slice/crio-e42f0748a48f01ab58db0d6baf6417b098a978752d7cc8558b83078051d7c6b1 WatchSource:0}: Error finding container e42f0748a48f01ab58db0d6baf6417b098a978752d7cc8558b83078051d7c6b1: Status 404 returned error can't find the container with id e42f0748a48f01ab58db0d6baf6417b098a978752d7cc8558b83078051d7c6b1
Oct 30 02:41:22.681304 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:22.681276    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea2f02d_9272_44dc_acd9_d899a2d711a9.slice/crio-86410e3fc7a266a6bd515d14c2d6e8fc36b8edef653874ba41f880dedba2c1e0 WatchSource:0}: Error finding container 86410e3fc7a266a6bd515d14c2d6e8fc36b8edef653874ba41f880dedba2c1e0: Status 404 returned error can't find the container with id 86410e3fc7a266a6bd515d14c2d6e8fc36b8edef653874ba41f880dedba2c1e0
Oct 30 02:41:22.987436 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.987399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:22.987628 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:22.987482    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:22.987628 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.987619    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:22.987741 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.987683    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:23.987663241 +0000 UTC m=+4.046323766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:22.988083 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.988060    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:22.988156 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:22.988112    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret podName:033b7d72-7e35-47ee-9cf9-d9993d21f4be nodeName:}" failed. No retries permitted until 2026-10-30 02:41:23.988097193 +0000 UTC m=+4.046757720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret") pod "global-pull-secret-syncer-6xh6w" (UID: "033b7d72-7e35-47ee-9cf9-d9993d21f4be") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:23.088049 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.087969    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:23.088205 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.088186    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:23.088265 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.088206    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:23.088265 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.088219    2571 projected.go:196] Error preparing data for projected volume kube-api-access-f276b for pod openshift-network-diagnostics/network-check-target-v7c87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:23.088378 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.088279    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b podName:41c6efff-036c-44b7-b0ff-700a1503ec82 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:24.088260977 +0000 UTC m=+4.146921489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f276b" (UniqueName: "kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b") pod "network-check-target-v7c87" (UID: "41c6efff-036c-44b7-b0ff-700a1503ec82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:23.346483 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.346317    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:36:22 +0000 UTC" deadline="2028-06-09 16:21:02.078906256 +0000 UTC"
Oct 30 02:41:23.346483 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.346380    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14125h39m38.73253046s"
Oct 30 02:41:23.402857 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.402364    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:23.402857 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.402495    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:23.419243 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.418409    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal" event={"ID":"adf118574a1cbe5f944e98682e2ff56b","Type":"ContainerStarted","Data":"4f950047b0f58b56268ceb2299d993c3c9f862b001011ae12f57e0851cda4ab1"}
Oct 30 02:41:23.424863 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.424754    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"86410e3fc7a266a6bd515d14c2d6e8fc36b8edef653874ba41f880dedba2c1e0"}
Oct 30 02:41:23.438541 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.438487    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" event={"ID":"d90d1fba-6d54-49b3-8792-36688f2242d3","Type":"ContainerStarted","Data":"392583615beca64131500f95cba5389c07e5ae636fb3e9e928c40e1fac6e4b29"}
Oct 30 02:41:23.448134 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.448098    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvqdb" event={"ID":"05e47302-4160-472b-988d-60585665c3bf","Type":"ContainerStarted","Data":"e62347cc65e9895ee37abee92e3e2c5ee559b1614ddce93bf56aa7c64255430d"}
Oct 30 02:41:23.456476 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.456427    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vj6lt" event={"ID":"fb013b42-f40c-4df9-bd10-ccb8975e54dc","Type":"ContainerStarted","Data":"4b5081c1421016df115012cff35da83e865949a3c9eed804eb2149d6c1f3159f"}
Oct 30 02:41:23.463965 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.463939    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mkxt8" event={"ID":"172aafae-6ad9-420e-a213-a8bc8f2439fe","Type":"ContainerStarted","Data":"b1d5c40431ac91b661df11beae61a7443d367df013cacb8dd5ce03f31796bd85"}
Oct 30 02:41:23.468791 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.468764    2571 generic.go:356] "Generic (PLEG): container finished" podID="a68cf21ce8d2089c87349d6b232245b0" containerID="62fcf0cd130d648fe8d4053f0e005f5a362504d9f030638a1e04787af6e3ae79" exitCode=0
Oct 30 02:41:23.468890 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.468845    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal" event={"ID":"a68cf21ce8d2089c87349d6b232245b0","Type":"ContainerDied","Data":"62fcf0cd130d648fe8d4053f0e005f5a362504d9f030638a1e04787af6e3ae79"}
Oct 30 02:41:23.489029 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.488961    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8l6sk" event={"ID":"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355","Type":"ContainerStarted","Data":"e42f0748a48f01ab58db0d6baf6417b098a978752d7cc8558b83078051d7c6b1"}
Oct 30 02:41:23.494175 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.494140    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg" event={"ID":"c27c447f-f1fa-4d0e-809e-f2945b673c32","Type":"ContainerStarted","Data":"0e9a5297aafc8d2b81006c842e17021e5f5bbe3490566423696e9b7d5631916c"}
Oct 30 02:41:23.506219 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.506142    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qb86g" event={"ID":"1628ebb0-5634-4a42-bca2-b09ecdbfd6be","Type":"ContainerStarted","Data":"0212d010014645cdc1d57804a89dd26b03fc3677c45a7b407b47bd8b578d709a"}
Oct 30 02:41:23.513654 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.513588    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k5blc" event={"ID":"818913a6-9255-4e66-b325-5ced6778ad64","Type":"ContainerStarted","Data":"46158511a4fd6b69bd5d6a1d126e3e938f9a98c1407c0cef949f17a93a53b875"}
Oct 30 02:41:23.520350 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.520152    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal" podStartSLOduration=2.520134948 podStartE2EDuration="2.520134948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:23.485706929 +0000 UTC m=+3.544367464" watchObservedRunningTime="2026-10-30 02:41:23.520134948 +0000 UTC m=+3.578795483"
Oct 30 02:41:23.997472 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.997385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:23.997472 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:23.997463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:23.997645 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.997559    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:23.997645 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.997590    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:23.997645 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.997636    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret podName:033b7d72-7e35-47ee-9cf9-d9993d21f4be nodeName:}" failed. No retries permitted until 2026-10-30 02:41:25.997616223 +0000 UTC m=+6.056276737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret") pod "global-pull-secret-syncer-6xh6w" (UID: "033b7d72-7e35-47ee-9cf9-d9993d21f4be") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:23.997759 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:23.997656    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:25.997646373 +0000 UTC m=+6.056306891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:24.098053 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:24.098017    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:24.098253 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:24.098223    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:24.098253 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:24.098247    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:24.098375 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:24.098261    2571 projected.go:196] Error preparing data for projected volume kube-api-access-f276b for pod openshift-network-diagnostics/network-check-target-v7c87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:24.098375 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:24.098318    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b podName:41c6efff-036c-44b7-b0ff-700a1503ec82 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:26.098300387 +0000 UTC m=+6.156960917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f276b" (UniqueName: "kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b") pod "network-check-target-v7c87" (UID: "41c6efff-036c-44b7-b0ff-700a1503ec82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:24.346986 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:24.346904    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:36:22 +0000 UTC" deadline="2028-06-04 06:45:33.654726168 +0000 UTC"
Oct 30 02:41:24.346986 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:24.346945    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13996h4m9.307786789s"
Oct 30 02:41:24.406328 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:24.405624    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:24.406328 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:24.405756    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:24.406328 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:24.406171    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:24.406328 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:24.406278    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:24.530074 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:24.530004    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal" event={"ID":"a68cf21ce8d2089c87349d6b232245b0","Type":"ContainerStarted","Data":"e5daf254c131b715ed7f719498977748c60428853db4e4d135882375c52fb328"}
Oct 30 02:41:25.402871 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:25.402395    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:25.402871 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:25.402535    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:25.546462 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:25.545042    2571 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="90333823464faa361ab4edff2aace20a9d9e02e638942e7e6518900b08a47a8b" exitCode=0
Oct 30 02:41:25.546462 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:25.546144    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" event={"ID":"d90d1fba-6d54-49b3-8792-36688f2242d3","Type":"ContainerDied","Data":"90333823464faa361ab4edff2aace20a9d9e02e638942e7e6518900b08a47a8b"}
Oct 30 02:41:25.605654 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:25.605595    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal" podStartSLOduration=4.605576176 podStartE2EDuration="4.605576176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:24.564566459 +0000 UTC m=+4.623227018" watchObservedRunningTime="2026-10-30 02:41:25.605576176 +0000 UTC m=+5.664236710"
Oct 30 02:41:26.016751 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:26.016609    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:26.016751 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:26.016677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:26.016950 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.016813    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:26.016950 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.016874    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.016856156 +0000 UTC m=+10.075516679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:26.017304 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.017281    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:26.017412 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.017365    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret podName:033b7d72-7e35-47ee-9cf9-d9993d21f4be nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.017325763 +0000 UTC m=+10.075986287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret") pod "global-pull-secret-syncer-6xh6w" (UID: "033b7d72-7e35-47ee-9cf9-d9993d21f4be") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:26.117879 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:26.117847    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:26.118061 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.118044    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:26.118124 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.118071    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:26.118124 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.118084    2571 projected.go:196] Error preparing data for projected volume kube-api-access-f276b for pod openshift-network-diagnostics/network-check-target-v7c87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:26.118236 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.118136    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b podName:41c6efff-036c-44b7-b0ff-700a1503ec82 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.118119573 +0000 UTC m=+10.176780088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f276b" (UniqueName: "kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b") pod "network-check-target-v7c87" (UID: "41c6efff-036c-44b7-b0ff-700a1503ec82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:26.402183 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:26.402104    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:26.402183 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:26.402149    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:26.402341 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.402258    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:26.402377 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:26.402329    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:26.553015 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:26.552299    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mkxt8" event={"ID":"172aafae-6ad9-420e-a213-a8bc8f2439fe","Type":"ContainerStarted","Data":"0a2dba7cddb0867455cae29840c15a14cebe95a4ae315aace462b8e0f21e5254"}
Oct 30 02:41:27.403413 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:27.403385    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:27.403568 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:27.403541    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:27.493841 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:27.493769    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:27.494721 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:27.494697    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:27.533000 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:27.532949    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mkxt8" podStartSLOduration=4.105774308 podStartE2EDuration="7.532928789s" totalImagesPullingTime="3.427154481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:26.594962978 +0000 UTC m=+6.653623512" watchObservedRunningTime="2026-10-30 02:41:27.532928789 +0000 UTC m=+7.591589323"
Oct 30 02:41:27.557135 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:27.557037    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qb86g" event={"ID":"1628ebb0-5634-4a42-bca2-b09ecdbfd6be","Type":"ContainerStarted","Data":"bf4925467f7acbc964ca6aeb655487eb22d7850d60c6a6cd586115e794e9dee5"}
Oct 30 02:41:28.401919 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:28.401840    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:28.402075 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:28.401992    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:28.402470 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:28.402438    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:28.402590 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:28.402566    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:28.561059 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:28.561033    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:28.561593 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:28.561487    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg" event={"ID":"c27c447f-f1fa-4d0e-809e-f2945b673c32","Type":"ContainerStarted","Data":"e623a281df16274c7f86f85a9845d4e714cb13c53fdfd348eedbec226f2bf308"}
Oct 30 02:41:29.403982 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:29.403328    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:29.403982 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:29.403482    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:29.545901 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:29.545868    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:41:29.567308 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:29.567254    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg" event={"ID":"c27c447f-f1fa-4d0e-809e-f2945b673c32","Type":"ContainerStarted","Data":"badf309793a2428fe4b3ba5b05c1e3a621233a16e6632ccb12fc18536e99123c"}
Oct 30 02:41:30.048615 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.048563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:30.048809 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.048642    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:30.048809 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.048768    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:30.048924 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.048833    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:38.048813599 +0000 UTC m=+18.107474111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:30.049298 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.049276    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:30.049397 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.049338    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret podName:033b7d72-7e35-47ee-9cf9-d9993d21f4be nodeName:}" failed. No retries permitted until 2026-10-30 02:41:38.049322757 +0000 UTC m=+18.107983276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret") pod "global-pull-secret-syncer-6xh6w" (UID: "033b7d72-7e35-47ee-9cf9-d9993d21f4be") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:30.149845 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.149758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:30.149995 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.149906    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:30.149995 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.149924    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:30.149995 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.149936    2571 projected.go:196] Error preparing data for projected volume kube-api-access-f276b for pod openshift-network-diagnostics/network-check-target-v7c87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:30.150156 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.149997    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b podName:41c6efff-036c-44b7-b0ff-700a1503ec82 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:38.149977415 +0000 UTC m=+18.208637940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f276b" (UniqueName: "kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b") pod "network-check-target-v7c87" (UID: "41c6efff-036c-44b7-b0ff-700a1503ec82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:30.326788 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.326676    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:41:29.545896411Z","UUID":"acb5ef24-ce11-4f6e-aaf3-83cddbd1dc31","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:41:30.331023 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.330998    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:41:30.331146 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.331034    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:41:30.405611 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.405583    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:30.405690 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.405631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:30.405762 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.405737    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:30.405854 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:30.405835    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:30.572644 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.572568    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k5blc" event={"ID":"818913a6-9255-4e66-b325-5ced6778ad64","Type":"ContainerStarted","Data":"edf5410bb387a557acb7186c1d9936150c76e1948dc9c7183b8e73bd5b7d9935"}
Oct 30 02:41:30.575610 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.575558    2571 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="5ff83304ce1d5bfc11112bf5dd255409c5b37a8d9bfb121dc053765be70950a6" exitCode=0
Oct 30 02:41:30.575712 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.575644    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" event={"ID":"d90d1fba-6d54-49b3-8792-36688f2242d3","Type":"ContainerDied","Data":"5ff83304ce1d5bfc11112bf5dd255409c5b37a8d9bfb121dc053765be70950a6"}
Oct 30 02:41:30.578009 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.577976    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvqdb" event={"ID":"05e47302-4160-472b-988d-60585665c3bf","Type":"ContainerStarted","Data":"4f92c59602e2c49fc66a63f4ba99648fa6fcaeeb2c346f8067e81682563c807f"}
Oct 30 02:41:30.609124 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.609078    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qb86g" podStartSLOduration=6.012994183 podStartE2EDuration="10.609063191s" totalImagesPullingTime="4.596069008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:27.591144009 +0000 UTC m=+7.649804545" watchObservedRunningTime="2026-10-30 02:41:30.609063191 +0000 UTC m=+10.667723724"
Oct 30 02:41:30.610091 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.610041    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k5blc" podStartSLOduration=2.96440818 podStartE2EDuration="10.610019731s" totalImagesPullingTime="7.645611551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:30.609695653 +0000 UTC m=+10.668356183" watchObservedRunningTime="2026-10-30 02:41:30.610019731 +0000 UTC m=+10.668680265"
Oct 30 02:41:30.690723 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:30.690457    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pvqdb" podStartSLOduration=3.531723223 podStartE2EDuration="10.690426773s" totalImagesPullingTime="7.15870355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:30.690318885 +0000 UTC m=+10.748979418" watchObservedRunningTime="2026-10-30 02:41:30.690426773 +0000 UTC m=+10.749087305"
Oct 30 02:41:31.402470 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:31.402379    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:31.402616 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:31.402538    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:31.582802 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:31.582770    2571 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="1d7e68d93539de3709e0d3d64b26f0fdc079e8e1d2e12b781da68dad2dac51ef" exitCode=0
Oct 30 02:41:31.583278 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:31.582874    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" event={"ID":"d90d1fba-6d54-49b3-8792-36688f2242d3","Type":"ContainerDied","Data":"1d7e68d93539de3709e0d3d64b26f0fdc079e8e1d2e12b781da68dad2dac51ef"}
Oct 30 02:41:31.586013 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:31.585987    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg" event={"ID":"c27c447f-f1fa-4d0e-809e-f2945b673c32","Type":"ContainerStarted","Data":"20d4715d3d596213563cf87b8111ffdeed31d42ad068c407f9bc5bfda24eccd3"}
Oct 30 02:41:32.402465 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:32.402418    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:32.402626 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:32.402433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:32.402626 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:32.402546    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:32.402719 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:32.402653    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:32.590112 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:32.590076    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8l6sk" event={"ID":"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355","Type":"ContainerStarted","Data":"c5fe62eb491d87f87a8ecf9578454b83535ccb309ab561ba23ac250769fb1acc"}
Oct 30 02:41:32.625566 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:32.625518    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg" podStartSLOduration=5.040996214 podStartE2EDuration="12.625504223s" totalImagesPullingTime="7.584508009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:31.668518201 +0000 UTC m=+11.727178735" watchObservedRunningTime="2026-10-30 02:41:32.625504223 +0000 UTC m=+12.684164766"
Oct 30 02:41:33.402242 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:33.402210    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:33.402429 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:33.402336    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:33.594922 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:33.594885    2571 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="dcad8297433ae34a8bf36551e1c0f147b2e1170c8f3869dcf108acc2b37d4a8d" exitCode=0
Oct 30 02:41:33.595545 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:33.594980    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" event={"ID":"d90d1fba-6d54-49b3-8792-36688f2242d3","Type":"ContainerDied","Data":"dcad8297433ae34a8bf36551e1c0f147b2e1170c8f3869dcf108acc2b37d4a8d"}
Oct 30 02:41:33.596562 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:33.596537    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vj6lt" event={"ID":"fb013b42-f40c-4df9-bd10-ccb8975e54dc","Type":"ContainerStarted","Data":"b5b2085b0b2ba1cf38e1c3a781854e5ba71b0c2cd65f1350ee4b398ffdcfed47"}
Oct 30 02:41:33.656029 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:33.655977    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8l6sk" podStartSLOduration=5.926427614 podStartE2EDuration="13.655963288s" totalImagesPullingTime="7.729535674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:32.626317593 +0000 UTC m=+12.684978125" watchObservedRunningTime="2026-10-30 02:41:33.655963288 +0000 UTC m=+13.714623820"
Oct 30 02:41:33.690694 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:33.690647    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vj6lt" podStartSLOduration=3.70376663 podStartE2EDuration="13.690629377s" totalImagesPullingTime="9.986862747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:33.689821017 +0000 UTC m=+13.748481555" watchObservedRunningTime="2026-10-30 02:41:33.690629377 +0000 UTC m=+13.749289916"
Oct 30 02:41:34.402802 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:34.402770    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:34.402968 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:34.402815    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:34.402968 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:34.402931    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:34.403347 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:34.403320    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:35.402675 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:35.402645    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:35.403271 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:35.402764    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:35.604322 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:35.604016    2571 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="f640f3d45bc40a7c7915c77913e520d77dc23b6d38d21459a12c5f8f56b652cf" exitCode=0
Oct 30 02:41:35.604322 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:35.604101    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" event={"ID":"d90d1fba-6d54-49b3-8792-36688f2242d3","Type":"ContainerDied","Data":"f640f3d45bc40a7c7915c77913e520d77dc23b6d38d21459a12c5f8f56b652cf"}
Oct 30 02:41:36.402077 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:36.402044    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:36.402077 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:36.402066    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:36.402286 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:36.402171    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:36.402286 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:36.402273    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:36.609307 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:36.609269    2571 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="4d29b2aba50ff427b30eee7815c84e07dd24265ec6cc436ac9fc2eb83426d516" exitCode=0
Oct 30 02:41:36.609673 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:36.609331    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" event={"ID":"d90d1fba-6d54-49b3-8792-36688f2242d3","Type":"ContainerDied","Data":"4d29b2aba50ff427b30eee7815c84e07dd24265ec6cc436ac9fc2eb83426d516"}
Oct 30 02:41:37.402985 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.402759    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:37.403180 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:37.403026    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:37.614595 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.614560    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"5452b85eb91201b817a09729cc62c78fc60a63a91bab2f0b604f11ad12742363"}
Oct 30 02:41:37.615063 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.614639    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"92437c9bae13edc1d38637c78a7050c175dd5718b5e48f7ebcaf98f0c10de5c8"}
Oct 30 02:41:37.615063 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.614683    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"e5c56e581dc0874a9b9b236fdd87a4ed8b2f7f138bfc06c603725ff5e47dc1ac"}
Oct 30 02:41:37.615063 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.614732    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"2bca3cc0cc623ad7840be9992f4e763a87caddf737faeacd8e793feb10745c29"}
Oct 30 02:41:37.615063 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.614781    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"96a9873b6e33c8089ce22a7b3424f62a7b6a41c3b41fbdfc8278e781804b967d"}
Oct 30 02:41:37.615063 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.614823    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"a4383bd6538b1f6e373eafbde5a7bec3231edae2412ee264c922da9a2b763ce6"}
Oct 30 02:41:37.617880 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.617852    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" event={"ID":"d90d1fba-6d54-49b3-8792-36688f2242d3","Type":"ContainerStarted","Data":"1b3964785d6f9809613cc9d4b8f03e76bf19f4a0dcf8da1b1e8f738e895f1d87"}
Oct 30 02:41:37.655254 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:37.655215    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hp7gt" podStartSLOduration=7.386369276 podStartE2EDuration="17.655203871s" totalImagesPullingTime="10.268834595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:37.655064891 +0000 UTC m=+17.713725423" watchObservedRunningTime="2026-10-30 02:41:37.655203871 +0000 UTC m=+17.713864402"
Oct 30 02:41:38.109004 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:38.108927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:38.109004 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:38.108969    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:38.109171 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.109056    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:38.109171 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.109056    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:38.109171 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.109102    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:54.109089986 +0000 UTC m=+34.167750502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:41:38.109171 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.109116    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret podName:033b7d72-7e35-47ee-9cf9-d9993d21f4be nodeName:}" failed. No retries permitted until 2026-10-30 02:41:54.109109875 +0000 UTC m=+34.167770385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret") pod "global-pull-secret-syncer-6xh6w" (UID: "033b7d72-7e35-47ee-9cf9-d9993d21f4be") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:41:38.209343 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:38.209312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:38.209480 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.209434    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:41:38.209480 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.209460    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:41:38.209480 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.209469    2571 projected.go:196] Error preparing data for projected volume kube-api-access-f276b for pod openshift-network-diagnostics/network-check-target-v7c87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:38.209593 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.209510    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b podName:41c6efff-036c-44b7-b0ff-700a1503ec82 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:54.209498453 +0000 UTC m=+34.268158964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f276b" (UniqueName: "kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b") pod "network-check-target-v7c87" (UID: "41c6efff-036c-44b7-b0ff-700a1503ec82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:41:38.402318 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:38.402247    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:38.402473 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.402355    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:38.402473 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:38.402408    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:38.402554 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:38.402516    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:39.402510 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:39.402459    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:39.402963 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:39.402589    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:39.624178 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:39.624104    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"25e47f34e7fe8e30cb8afa326a633311a15981a87e178849c50470b10a1e4bf6"}
Oct 30 02:41:40.403121 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:40.403085    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:40.403745 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:40.403176    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:40.403745 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:40.403253    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:40.403745 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:40.403336    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:41.402772 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:41.402739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:41.402935 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:41.402843    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:41.631113 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:41.631028    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" event={"ID":"cea2f02d-9272-44dc-acd9-d899a2d711a9","Type":"ContainerStarted","Data":"4f115006cc9c673901681ac9c4e692157a48a8caf85af0adfe86406614eafa37"}
Oct 30 02:41:41.631843 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:41.631229    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:41.631843 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:41.631297    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:41.646236 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:41.646211    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:41.649850 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:41.649835    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:41.705199 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:41.705157    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj" podStartSLOduration=7.5083062080000005 podStartE2EDuration="21.705145819s" totalImagesPullingTime="14.196839611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:41.70442592 +0000 UTC m=+21.763086453" watchObservedRunningTime="2026-10-30 02:41:41.705145819 +0000 UTC m=+21.763806351"
Oct 30 02:41:42.401931 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:42.401906    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:42.401931 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:42.401937    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:42.402119 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:42.402016    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:42.402204 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:42.402162    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:42.632960 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:42.632935    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:43.398845 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:43.398748    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v7c87"]
Oct 30 02:41:43.398991 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:43.398846    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:43.398991 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:43.398938    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:43.402367 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:43.402343    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gztvh"]
Oct 30 02:41:43.402494 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:43.402432    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:43.402561 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:43.402541    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:43.408873 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:43.408852    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6xh6w"]
Oct 30 02:41:43.408953 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:43.408920    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:43.409001 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:43.408981    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:43.635336 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:43.635312    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:45.402094 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:45.402069    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:45.402374 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:45.402134    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:45.402374 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:45.402228    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:45.402374 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:45.402232    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:45.402488 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:45.402378    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:45.402523 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:45.402483    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:46.435146 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:46.435118    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:46.435548 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:46.435235    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:46.435776 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:46.435757    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:41:47.401972 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:47.401932    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:47.401972 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:47.401970    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:47.401972 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:47.401976    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:47.402164 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:47.402050    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:47.402281 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:47.402168    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:47.402281 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:47.402243    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:48.053910 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:48.053881    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:48.054350 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:48.054070    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:41:48.064979 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:48.064957    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:41:49.401774 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.401743    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:49.401774 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.401759    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:49.402163 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.401844    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:49.402163 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:49.401854    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gztvh" podUID="cf492558-e351-45c6-8772-2324e53b4a97"
Oct 30 02:41:49.402163 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:49.401904    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c87" podUID="41c6efff-036c-44b7-b0ff-700a1503ec82"
Oct 30 02:41:49.402163 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:49.401974    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xh6w" podUID="033b7d72-7e35-47ee-9cf9-d9993d21f4be"
Oct 30 02:41:49.689926 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.689904    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeReady"
Oct 30 02:41:49.690059 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.690008    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:41:49.737914 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.737886    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8k7nc"]
Oct 30 02:41:49.741194 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.741165    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.741319 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.741165    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5tmj8"]
Oct 30 02:41:49.744251 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.744233    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:41:49.753903 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.753887    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:41:49.754037 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.753924    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:41:49.754091 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.754080    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hbwgl\""
Oct 30 02:41:49.758931 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.758912    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8k7nc"]
Oct 30 02:41:49.764919 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.764900    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5tmj8"]
Oct 30 02:41:49.854602 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.854578    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:41:49.854731 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.854684    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:41:49.854795 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.854745    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:41:49.854828 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.854818    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcz9c\""
Oct 30 02:41:49.896988 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.896963    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswjr\" (UniqueName: \"kubernetes.io/projected/1436ee1e-5bbf-488b-812b-92c7bc0906a2-kube-api-access-bswjr\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.897075 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.896990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1436ee1e-5bbf-488b-812b-92c7bc0906a2-tmp-dir\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.897141 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.897111    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c080eab2-cfb6-4495-ba04-475ad7344a43-cert\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:41:49.897198 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.897171    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgmrq\" (UniqueName: \"kubernetes.io/projected/c080eab2-cfb6-4495-ba04-475ad7344a43-kube-api-access-mgmrq\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:41:49.897255 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.897196    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.897255 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.897238    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1436ee1e-5bbf-488b-812b-92c7bc0906a2-config-volume\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.998396 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.998337    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c080eab2-cfb6-4495-ba04-475ad7344a43-cert\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:41:49.998396 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.998383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgmrq\" (UniqueName: \"kubernetes.io/projected/c080eab2-cfb6-4495-ba04-475ad7344a43-kube-api-access-mgmrq\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:41:49.998584 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.998409    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.998584 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.998433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1436ee1e-5bbf-488b-812b-92c7bc0906a2-config-volume\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.998584 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.998491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswjr\" (UniqueName: \"kubernetes.io/projected/1436ee1e-5bbf-488b-812b-92c7bc0906a2-kube-api-access-bswjr\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.998584 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.998519    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1436ee1e-5bbf-488b-812b-92c7bc0906a2-tmp-dir\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.998584 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:49.998551    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:49.998807 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:49.998617    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:50.498602224 +0000 UTC m=+30.557262733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:49.998890 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.998873    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1436ee1e-5bbf-488b-812b-92c7bc0906a2-tmp-dir\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:49.999083 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:49.999062    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1436ee1e-5bbf-488b-812b-92c7bc0906a2-config-volume\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:50.002171 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:50.002154    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c080eab2-cfb6-4495-ba04-475ad7344a43-cert\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:41:50.018113 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:50.018087    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswjr\" (UniqueName: \"kubernetes.io/projected/1436ee1e-5bbf-488b-812b-92c7bc0906a2-kube-api-access-bswjr\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:50.018597 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:50.018582    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgmrq\" (UniqueName: \"kubernetes.io/projected/c080eab2-cfb6-4495-ba04-475ad7344a43-kube-api-access-mgmrq\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:41:50.059613 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:50.059589    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:41:50.426083 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:50.425922    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5tmj8"]
Oct 30 02:41:50.431119 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:50.431094    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc080eab2_cfb6_4495_ba04_475ad7344a43.slice/crio-30404c8c7b2776516c0bd0753098edaf124eec562767c1b5995e4af673707d83 WatchSource:0}: Error finding container 30404c8c7b2776516c0bd0753098edaf124eec562767c1b5995e4af673707d83: Status 404 returned error can't find the container with id 30404c8c7b2776516c0bd0753098edaf124eec562767c1b5995e4af673707d83
Oct 30 02:41:50.503126 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:50.503089    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:50.503226 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:50.503217    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:50.503285 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:50.503275    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:51.50325739 +0000 UTC m=+31.561917902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:50.650785 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:50.650753    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5tmj8" event={"ID":"c080eab2-cfb6-4495-ba04-475ad7344a43","Type":"ContainerStarted","Data":"30404c8c7b2776516c0bd0753098edaf124eec562767c1b5995e4af673707d83"}
Oct 30 02:41:51.402717 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.402681    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:51.402900 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.402700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:51.402900 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.402803    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:51.415987 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.415959    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j74dx\""
Oct 30 02:41:51.415987 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.415973    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:41:51.426150 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.426128    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:41:51.426597 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.426154    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-czntg\""
Oct 30 02:41:51.426597 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.426297    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:41:51.426597 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.426317    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d89pf\""
Oct 30 02:41:51.426597 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.426356    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:41:51.509769 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:51.509738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:51.509908 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:51.509876    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:51.509973 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:51.509944    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:53.509923487 +0000 UTC m=+33.568584002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:53.525348 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:53.525315    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:53.525976 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:53.525489    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:53.525976 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:53.525552    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:57.525538223 +0000 UTC m=+37.584198732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:53.659842 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:53.659810    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5tmj8" event={"ID":"c080eab2-cfb6-4495-ba04-475ad7344a43","Type":"ContainerStarted","Data":"412e196d43899faea2cdc74c0c92c17686668cbdedac5b739258c4b9c61b9712"}
Oct 30 02:41:53.696397 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:53.696354    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5tmj8" podStartSLOduration=2.326372491 podStartE2EDuration="4.696339476s" totalImagesPullingTime="2.369966985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:53.695366522 +0000 UTC m=+33.754027057" watchObservedRunningTime="2026-10-30 02:41:53.696339476 +0000 UTC m=+33.755000008"
Oct 30 02:41:54.128831 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.128796    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:54.128991 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.128843    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:41:54.128991 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.128941    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:41:54.128991 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.128992    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:26.128980003 +0000 UTC m=+66.187640527 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : secret "metrics-daemon-secret" not found
Oct 30 02:41:54.131996 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.131964    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:54.150849 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.150823    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bvj8c"]
Oct 30 02:41:54.174334 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.172666    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:41:54.199278 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.199255    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bvj8c"]
Oct 30 02:41:54.199392 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.199371    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.217836 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.217814    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jbbb7-m2nlh"]
Oct 30 02:41:54.229700 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.229677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:54.231671 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.231502    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.232089 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.232069    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:54.238076 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.238056    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jbbb7-m2nlh"]
Oct 30 02:41:54.282677 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.282654    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:41:54.282784 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.282681    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8z95j\""
Oct 30 02:41:54.282995 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.282979    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:41:54.283051 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.283036    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7d7z9\""
Oct 30 02:41:54.283114 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.283102    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:41:54.283143 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.283119    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:41:54.283143 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.283139    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:41:54.303622 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.303605    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:41:54.330792 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.330772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.330845 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.330813    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180f424e-5860-452d-98b6-c0d4585dd13c-crio-socket\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.330845 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.330840    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfszg\" (UniqueName: \"kubernetes.io/projected/180f424e-5860-452d-98b6-c0d4585dd13c-kube-api-access-cfszg\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.330907 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.330886    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.330998 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.330984    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.331033 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.331010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.331064 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.331032    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.331064 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.331053    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180f424e-5860-452d-98b6-c0d4585dd13c-data-volume\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.331120 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.331110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqnr\" (UniqueName: \"kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.432161 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432138    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180f424e-5860-452d-98b6-c0d4585dd13c-data-volume\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.432266 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432169    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqnr\" (UniqueName: \"kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.432266 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.432266 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432246    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180f424e-5860-452d-98b6-c0d4585dd13c-crio-socket\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.432390 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432279    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfszg\" (UniqueName: \"kubernetes.io/projected/180f424e-5860-452d-98b6-c0d4585dd13c-kube-api-access-cfszg\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.432390 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.432390 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432332    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.432606 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.432491    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:54.432606 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432530    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180f424e-5860-452d-98b6-c0d4585dd13c-crio-socket\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.432606 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.432584    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:41:54.932551383 +0000 UTC m=+34.991211910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:54.432755 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432602    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180f424e-5860-452d-98b6-c0d4585dd13c-data-volume\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.432755 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.432755 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432602    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.432755 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.432755 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.432724    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:41:54.432919 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.432784    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:41:54.932772408 +0000 UTC m=+34.991432917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : configmap "kube-rbac-proxy" not found
Oct 30 02:41:54.433020 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.432999    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.433415 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.433397    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.450985 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.450963    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfszg\" (UniqueName: \"kubernetes.io/projected/180f424e-5860-452d-98b6-c0d4585dd13c-kube-api-access-cfszg\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.451099 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.451087    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqnr\" (UniqueName: \"kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.485609 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.485587    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:54.494906 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.494880    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6xh6w"]
Oct 30 02:41:54.499512 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:54.499488    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033b7d72_7e35_47ee_9cf9_d9993d21f4be.slice/crio-43fb03925b67033cf9a96d2960dd54a9b7b358f4b165a03405c61473744b59cd WatchSource:0}: Error finding container 43fb03925b67033cf9a96d2960dd54a9b7b358f4b165a03405c61473744b59cd: Status 404 returned error can't find the container with id 43fb03925b67033cf9a96d2960dd54a9b7b358f4b165a03405c61473744b59cd
Oct 30 02:41:54.574128 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.574101    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:41:54.675263 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.675229    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6xh6w" event={"ID":"033b7d72-7e35-47ee-9cf9-d9993d21f4be","Type":"ContainerStarted","Data":"43fb03925b67033cf9a96d2960dd54a9b7b358f4b165a03405c61473744b59cd"}
Oct 30 02:41:54.799669 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.799608    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v7c87"]
Oct 30 02:41:54.804917 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:54.804892    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c6efff_036c_44b7_b0ff_700a1503ec82.slice/crio-283a49d923721732fdec0697ba412765c03a3fb0f69fe586148ebfa90e3dd311 WatchSource:0}: Error finding container 283a49d923721732fdec0697ba412765c03a3fb0f69fe586148ebfa90e3dd311: Status 404 returned error can't find the container with id 283a49d923721732fdec0697ba412765c03a3fb0f69fe586148ebfa90e3dd311
Oct 30 02:41:54.889768 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.889740    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jbbb7-m2nlh"]
Oct 30 02:41:54.894204 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:54.894179    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5264ed23_be36_4ef6_bdf7_3d694ab4ec19.slice/crio-c2bd19973fbd0146c464885e597731726138d9a4cc61a2a0d651c1559a9e8e3c WatchSource:0}: Error finding container c2bd19973fbd0146c464885e597731726138d9a4cc61a2a0d651c1559a9e8e3c: Status 404 returned error can't find the container with id c2bd19973fbd0146c464885e597731726138d9a4cc61a2a0d651c1559a9e8e3c
Oct 30 02:41:54.936611 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.936584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.936699 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:54.936629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:54.936746 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.936713    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:41:54.936746 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.936739    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:54.936809 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.936776    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:41:55.936758679 +0000 UTC m=+35.995419205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : configmap "kube-rbac-proxy" not found
Oct 30 02:41:54.936809 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:54.936791    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:41:55.936785247 +0000 UTC m=+35.995445757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:55.160728 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.160695    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"]
Oct 30 02:41:55.178891 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.178870    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"
Oct 30 02:41:55.186405 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.186350    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"]
Oct 30 02:41:55.193585 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.193567    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tbftn\""
Oct 30 02:41:55.193716 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.193627    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:41:55.193716 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.193654    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:41:55.339772 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.339733    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvs8\" (UniqueName: \"kubernetes.io/projected/289d5912-6d50-4d1a-b359-1db38fd88f08-kube-api-access-hqvs8\") pod \"migrator-66df5d496b-z89rx\" (UID: \"289d5912-6d50-4d1a-b359-1db38fd88f08\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"
Oct 30 02:41:55.434055 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.433983    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xhrrk"]
Oct 30 02:41:55.440168 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.440135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvs8\" (UniqueName: \"kubernetes.io/projected/289d5912-6d50-4d1a-b359-1db38fd88f08-kube-api-access-hqvs8\") pod \"migrator-66df5d496b-z89rx\" (UID: \"289d5912-6d50-4d1a-b359-1db38fd88f08\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"
Oct 30 02:41:55.449903 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.449880    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.457310 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.455613    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xhrrk"]
Oct 30 02:41:55.459945 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.459905    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvs8\" (UniqueName: \"kubernetes.io/projected/289d5912-6d50-4d1a-b359-1db38fd88f08-kube-api-access-hqvs8\") pod \"migrator-66df5d496b-z89rx\" (UID: \"289d5912-6d50-4d1a-b359-1db38fd88f08\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"
Oct 30 02:41:55.463294 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.462424    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 02:41:55.463294 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.462898    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 02:41:55.463294 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.463087    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-blzpj\""
Oct 30 02:41:55.463294 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.463159    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 02:41:55.463294 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.463259    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 02:41:55.504038 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.504013    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"
Oct 30 02:41:55.540898 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.540870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sln\" (UniqueName: \"kubernetes.io/projected/b582586d-a43f-4e66-b5c2-2d83f406f0c3-kube-api-access-l4sln\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.541002 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.540946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.541040 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.541009    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-key\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.641825 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.641787    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sln\" (UniqueName: \"kubernetes.io/projected/b582586d-a43f-4e66-b5c2-2d83f406f0c3-kube-api-access-l4sln\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.641825 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.641836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.642499 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.641875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-key\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.643143 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.643107    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.646953 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.646928    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-key\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.661230 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.661182    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sln\" (UniqueName: \"kubernetes.io/projected/b582586d-a43f-4e66-b5c2-2d83f406f0c3-kube-api-access-l4sln\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.679719 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.679692    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v7c87" event={"ID":"41c6efff-036c-44b7-b0ff-700a1503ec82","Type":"ContainerStarted","Data":"283a49d923721732fdec0697ba412765c03a3fb0f69fe586148ebfa90e3dd311"}
Oct 30 02:41:55.682111 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.682057    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh" event={"ID":"5264ed23-be36-4ef6-bdf7-3d694ab4ec19","Type":"ContainerStarted","Data":"c2bd19973fbd0146c464885e597731726138d9a4cc61a2a0d651c1559a9e8e3c"}
Oct 30 02:41:55.777902 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.777662    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:41:55.819604 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.819525    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"]
Oct 30 02:41:55.827296 ip-10-0-133-197 kubenswrapper[2571]: W1030 02:41:55.826975    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289d5912_6d50_4d1a_b359_1db38fd88f08.slice/crio-ce4a54897febdc3b7bfeb7d0455ddad06b1a91dad52d93ba1f60d6da39c06928 WatchSource:0}: Error finding container ce4a54897febdc3b7bfeb7d0455ddad06b1a91dad52d93ba1f60d6da39c06928: Status 404 returned error can't find the container with id ce4a54897febdc3b7bfeb7d0455ddad06b1a91dad52d93ba1f60d6da39c06928
Oct 30 02:41:55.944220 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.944176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:55.944369 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:55.944246    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:55.944369 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:55.944346    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:41:55.944369 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:55.944360    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:55.945662 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:55.944409    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:41:57.944390459 +0000 UTC m=+38.003050975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : configmap "kube-rbac-proxy" not found
Oct 30 02:41:55.945662 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:55.944429    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:41:57.944419153 +0000 UTC m=+38.003079663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:56.145759 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:56.145683    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xhrrk"]
Oct 30 02:41:56.686802 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:56.686755    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk" event={"ID":"b582586d-a43f-4e66-b5c2-2d83f406f0c3","Type":"ContainerStarted","Data":"acea34652281e42a25419e91924f50b06f6b1529982d1e4dbd016dfc6334cc78"}
Oct 30 02:41:56.689611 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:56.689545    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx" event={"ID":"289d5912-6d50-4d1a-b359-1db38fd88f08","Type":"ContainerStarted","Data":"ce4a54897febdc3b7bfeb7d0455ddad06b1a91dad52d93ba1f60d6da39c06928"}
Oct 30 02:41:57.559667 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:57.559624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:41:57.559853 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:57.559829    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:41:57.559930 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:57.559893    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:05.559875209 +0000 UTC m=+45.618535722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:41:57.964180 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:57.963607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:57.964180 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:57.963739    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:57.964180 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:57.964071    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:41:57.964180 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:57.964088    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.964063954 +0000 UTC m=+42.022724478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:41:57.964180 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:57.964146    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:41:57.964971 ip-10-0-133-197 kubenswrapper[2571]: E1030 02:41:57.964205    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:42:01.964187723 +0000 UTC m=+42.022848253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : configmap "kube-rbac-proxy" not found
Oct 30 02:41:58.698974 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:58.698599    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx" event={"ID":"289d5912-6d50-4d1a-b359-1db38fd88f08","Type":"ContainerStarted","Data":"be030f58124e7f938a9d6fe48b112fea5577032ded1b384458b93f108b0d34a2"}
Oct 30 02:41:58.700776 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:58.700749    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh" event={"ID":"5264ed23-be36-4ef6-bdf7-3d694ab4ec19","Type":"ContainerStarted","Data":"e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74"}
Oct 30 02:41:58.734710 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:58.734647    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx" podStartSLOduration=1.87098356 podStartE2EDuration="3.734627564s" totalImagesPullingTime="1.863644004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:58.733505853 +0000 UTC m=+38.792166385" watchObservedRunningTime="2026-10-30 02:41:58.734627564 +0000 UTC m=+38.793288097"
Oct 30 02:41:58.769525 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:58.769466    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh" podStartSLOduration=1.285824693 podStartE2EDuration="4.769436699s" totalImagesPullingTime="3.483612006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:58.768616504 +0000 UTC m=+38.827277039" watchObservedRunningTime="2026-10-30 02:41:58.769436699 +0000 UTC m=+38.828097231"
Oct 30 02:41:59.705880 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:59.705844    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v7c87" event={"ID":"41c6efff-036c-44b7-b0ff-700a1503ec82","Type":"ContainerStarted","Data":"fac4a2b201d5eda6bcccbbf3fe1ae5f4e101194513dd7db42b64bce807c81824"}
Oct 30 02:41:59.706661 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:59.705938    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:41:59.708280 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:59.708227    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk" event={"ID":"b582586d-a43f-4e66-b5c2-2d83f406f0c3","Type":"ContainerStarted","Data":"5e10987bc4188b55c7525e050d5d1e446b235c6e72bd25e9ac1f1b26797cb1ec"}
Oct 30 02:41:59.741104 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:59.741004    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v7c87" podStartSLOduration=35.52257376 podStartE2EDuration="39.740987408s" totalImagesPullingTime="4.218413648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:59.74075252 +0000 UTC m=+39.799413049" watchObservedRunningTime="2026-10-30 02:41:59.740987408 +0000 UTC m=+39.799647940"
Oct 30 02:41:59.790596 ip-10-0-133-197 kubenswrapper[2571]: I1030 02:41:59.790542    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk" podStartSLOduration=1.470885481 podStartE2EDuration="4.790525641s" totalImagesPullingTime="3.31964016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:59.789688572 +0000 UTC m=+39.848349104" watchObservedRunningTime="2026-10-30 02:41:59.790525641 +0000 UTC m=+39.849186174"
Oct 30 02:42:00.923987 ip-10-0-133-197 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:42:00.938133 ip-10-0-133-197 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:42:00.938301 ip-10-0-133-197 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:42:00.938483 ip-10-0-133-197 systemd[1]: kubelet.service: Consumed 2.343s CPU time.
Oct 30 02:42:00.948977 ip-10-0-133-197 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:42:01.074485 ip-10-0-133-197 kubenswrapper[4334]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:42:01.074485 ip-10-0-133-197 kubenswrapper[4334]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:42:01.074485 ip-10-0-133-197 kubenswrapper[4334]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:42:01.074485 ip-10-0-133-197 kubenswrapper[4334]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:42:01.076589 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076502    4334 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:42:01.076589 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076589    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076593    4334 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076596    4334 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076599    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076602    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076605    4334 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076608    4334 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076610    4334 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076613    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076616    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076619    4334 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076621    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076626    4334 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076630    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076634    4334 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076639    4334 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076642    4334 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076646    4334 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076649    4334 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:42:01.076662 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076652    4334 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076655    4334 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076659    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076662    4334 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076665    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076668    4334 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076671    4334 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076673    4334 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076676    4334 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076679    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076682    4334 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076685    4334 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076687    4334 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076690    4334 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076692    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076694    4334 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076697    4334 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076700    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076702    4334 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076704    4334 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:42:01.077133 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076707    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076710    4334 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076713    4334 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076715    4334 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076718    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076721    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076725    4334 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076727    4334 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076730    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076733    4334 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076735    4334 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076738    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076741    4334 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076743    4334 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076746    4334 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076748    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076751    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076754    4334 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076756    4334 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:42:01.077673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076759    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076761    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076764    4334 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076766    4334 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076769    4334 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076771    4334 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076774    4334 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076776    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076779    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076781    4334 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076784    4334 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076787    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076789    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076792    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076794    4334 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076797    4334 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076800    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076803    4334 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076805    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:42:01.078157 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076809    4334 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076811    4334 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076814    4334 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076816    4334 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076819    4334 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076821    4334 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076824    4334 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076827    4334 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076829    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076832    4334 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076834    4334 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076837    4334 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076839    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076842    4334 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076844    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076847    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076850    4334 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076852    4334 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076855    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076857    4334 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:42:01.078641 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076860    4334 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076862    4334 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076865    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076867    4334 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076870    4334 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076873    4334 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076877    4334 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.076879    4334 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077633    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077640    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077643    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077646    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077649    4334 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077652    4334 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077655    4334 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077657    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077660    4334 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077663    4334 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077665    4334 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:42:01.079131 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077668    4334 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077670    4334 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077673    4334 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077676    4334 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077679    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077682    4334 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077684    4334 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077687    4334 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077690    4334 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077692    4334 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077695    4334 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077697    4334 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077700    4334 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077703    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077706    4334 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077708    4334 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077711    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077714    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077716    4334 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077720    4334 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:42:01.079618 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077723    4334 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077725    4334 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077728    4334 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077730    4334 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077733    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077736    4334 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077738    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077742    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077745    4334 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077747    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077750    4334 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077753    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077755    4334 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077758    4334 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077760    4334 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077763    4334 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077765    4334 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077768    4334 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077771    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077773    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:42:01.080100 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077776    4334 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077780    4334 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077783    4334 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077786    4334 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077789    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077792    4334 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077795    4334 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077798    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077801    4334 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077804    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077807    4334 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077810    4334 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077812    4334 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077815    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077818    4334 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077820    4334 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077823    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077825    4334 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077828    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:42:01.080605 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077830    4334 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077833    4334 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077836    4334 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077838    4334 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077841    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077843    4334 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077846    4334 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077848    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077851    4334 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077854    4334 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077856    4334 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077860    4334 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077862    4334 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077865    4334 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077867    4334 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077870    4334 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077872    4334 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077875    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077877    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:42:01.081070 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077880    4334 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077882    4334 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077888    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077891    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077894    4334 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077896    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077899    4334 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077901    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077905    4334 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077910    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077913    4334 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077916    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077920    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077923    4334 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077926    4334 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077929    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.077932    4334 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078424    4334 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078430    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:42:01.081545 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078434    4334 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078437    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078440    4334 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078457    4334 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078460    4334 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078463    4334 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078466    4334 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078469    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078471    4334 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078474    4334 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078477    4334 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078479    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078483    4334 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078487    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078490    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078494    4334 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078497    4334 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078499    4334 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078502    4334 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:42:01.082045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078504    4334 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078507    4334 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078510    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078512    4334 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078515    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078517    4334 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078520    4334 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078522    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078526    4334 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078530    4334 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078533    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078536    4334 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078538    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078541    4334 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078543    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078546    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078548    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078551    4334 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078553    4334 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:42:01.082547 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078556    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078558    4334 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078561    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078565    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078568    4334 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078570    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078573    4334 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078575    4334 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078578    4334 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078581    4334 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078584    4334 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078586    4334 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078589    4334 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078591    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078594    4334 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078597    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078599    4334 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078602    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078604    4334 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:42:01.083045 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078607    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078610    4334 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078612    4334 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078615    4334 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078618    4334 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078620    4334 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078623    4334 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078625    4334 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078628    4334 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078631    4334 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078634    4334 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078636    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078639    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078641    4334 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078644    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078647    4334 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078650    4334 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078653    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078656    4334 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078658    4334 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:42:01.083523 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078661    4334 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078663    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078666    4334 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078668    4334 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078671    4334 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078674    4334 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078676    4334 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078679    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078681    4334 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078683    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078686    4334 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078689    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078691    4334 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078694    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078696    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078699    4334 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078702    4334 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078705    4334 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078708    4334 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:42:01.084030 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078711    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078714    4334 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078716    4334 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078719    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078721    4334 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078724    4334 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078726    4334 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.078729    4334 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078845    4334 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078866    4334 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078879    4334 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078886    4334 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078892    4334 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078896    4334 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078900    4334 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078905    4334 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078908    4334 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078911    4334 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078914    4334 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078917    4334 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078921    4334 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078924    4334 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078927    4334 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:42:01.084610 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078933    4334 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078936    4334 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078939    4334 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078944    4334 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078947    4334 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078950    4334 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078953    4334 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078960    4334 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078965    4334 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078969    4334 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078972    4334 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078976    4334 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078979    4334 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078982    4334 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078985    4334 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078988    4334 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078991    4334 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078996    4334 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.078999    4334 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079003    4334 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079006    4334 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079009    4334 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079012    4334 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079016    4334 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079019    4334 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:42:01.085324 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079022    4334 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079026    4334 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079029    4334 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079033    4334 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079036    4334 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079039    4334 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079043    4334 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079048    4334 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079051    4334 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079055    4334 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079058    4334 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079061    4334 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079064    4334 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079067    4334 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079072    4334 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079076    4334 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079079    4334 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079083    4334 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079086    4334 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079089    4334 flags.go:64] FLAG: --help="false"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079092    4334 flags.go:64] FLAG: --hostname-override="ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079096    4334 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079099    4334 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:42:01.085955 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079102    4334 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079105    4334 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079109    4334 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079112    4334 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079115    4334 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079118    4334 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079121    4334 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079124    4334 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079127    4334 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079131    4334 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079135    4334 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079138    4334 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079141    4334 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079144    4334 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079147    4334 flags.go:64] FLAG: --lock-file=""
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079150    4334 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079154    4334 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079158    4334 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079164    4334 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079167    4334 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079170    4334 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079173    4334 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079176    4334 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079181    4334 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:42:01.086543 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079184    4334 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079187    4334 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079192    4334 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079196    4334 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079200    4334 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079203    4334 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079206    4334 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079209    4334 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079213    4334 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079216    4334 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079219    4334 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079222    4334 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079229    4334 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079233    4334 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079236    4334 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079239    4334 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079242    4334 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079245    4334 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079248    4334 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079251    4334 flags.go:64] FLAG: --port="10250"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079255    4334 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079258    4334 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-006b16d6b6dbf3084"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079261    4334 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079264    4334 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079268    4334 flags.go:64] FLAG: --register-node="true"
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079271    4334 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:42:01.087230 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079275    4334 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079278    4334 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079281    4334 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079284    4334 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079288    4334 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079293    4334 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079296    4334 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079299    4334 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079302    4334 flags.go:64] FLAG: --runonce="false"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079310    4334 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079313    4334 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079317    4334 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079320    4334 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079323    4334 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079326    4334 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079329    4334 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079333    4334 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079337    4334 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079340    4334 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079343    4334 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079346    4334 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079349    4334 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079352    4334 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079355    4334 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079361    4334 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:42:01.087891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079364    4334 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079367    4334 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079372    4334 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079375    4334 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079378    4334 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079382    4334 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079386    4334 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079389    4334 flags.go:64] FLAG: --v="2"
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079393    4334 flags.go:64] FLAG: --version="false"
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079397    4334 flags.go:64] FLAG: --vmodule=""
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079402    4334 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079405    4334 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079559    4334 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079565    4334 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079568    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079572    4334 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079574    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079578    4334 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079582    4334 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079586    4334 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079590    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079592    4334 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:42:01.088659 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079595    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079598    4334 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079600    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079603    4334 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079606    4334 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079609    4334 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079611    4334 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079614    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079617    4334 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079619    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079622    4334 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079625    4334 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079628    4334 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079631    4334 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079633    4334 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079637    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079640    4334 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079642    4334 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079645    4334 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079647    4334 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:42:01.089197 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079650    4334 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079652    4334 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079657    4334 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079660    4334 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079662    4334 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079664    4334 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079667    4334 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079670    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079673    4334 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079675    4334 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079678    4334 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079680    4334 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079683    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079685    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079688    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079690    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079693    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079695    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079704    4334 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079707    4334 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:42:01.089764 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079710    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079712    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079715    4334 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079717    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079720    4334 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079723    4334 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079725    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079728    4334 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079731    4334 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079733    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079736    4334 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079738    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079741    4334 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079743    4334 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079747    4334 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079750    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079753    4334 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079756    4334 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079760    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:42:01.090342 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079763    4334 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079766    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079775    4334 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079778    4334 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079780    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079783    4334 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079785    4334 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079788    4334 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079790    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079793    4334 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079795    4334 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079798    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079801    4334 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079803    4334 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079806    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079809    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079812    4334 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079814    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079817    4334 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079819    4334 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:42:01.091125 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079822    4334 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079825    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079827    4334 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079830    4334 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079832    4334 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079835    4334 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079837    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079841    4334 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079843    4334 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079846    4334 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079849    4334 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079851    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079854    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079856    4334 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079859    4334 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079862    4334 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:42:01.091913 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079864    4334 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.079891    4334 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079931    4334 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079935    4334 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079938    4334 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079941    4334 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079944    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079947    4334 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079950    4334 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079952    4334 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079955    4334 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079958    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079960    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079963    4334 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079966    4334 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079968    4334 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:42:01.092332 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079971    4334 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079973    4334 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079976    4334 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079978    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079981    4334 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079984    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079987    4334 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079992    4334 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079996    4334 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.079999    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080002    4334 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080004    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080007    4334 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080010    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080012    4334 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080015    4334 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080018    4334 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080021    4334 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080023    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:42:01.092747 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080027    4334 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080030    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080032    4334 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080035    4334 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080037    4334 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080040    4334 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080042    4334 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080045    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080047    4334 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080050    4334 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080053    4334 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080055    4334 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080058    4334 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080060    4334 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080063    4334 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080065    4334 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080068    4334 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080070    4334 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080073    4334 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080076    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:42:01.093205 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080079    4334 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080082    4334 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080085    4334 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080087    4334 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080090    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080092    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080095    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080098    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080101    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080103    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080106    4334 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080109    4334 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080113    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080116    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080118    4334 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080121    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080137    4334 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080141    4334 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080144    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080147    4334 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:42:01.093726 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080150    4334 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080152    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080155    4334 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080157    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080160    4334 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080162    4334 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080165    4334 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080168    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080170    4334 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080176    4334 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080180    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080183    4334 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080187    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080190    4334 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080192    4334 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080195    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080198    4334 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080200    4334 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080203    4334 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:42:01.094266 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080206    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080208    4334 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080211    4334 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080214    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080217    4334 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080221    4334 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080224    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080227    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080230    4334 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080232    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080235    4334 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080237    4334 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080241    4334 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.080243    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.085561    4334 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.085575    4334 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.085592    4334 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.085597    4334 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085649    4334 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:42:01.094798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085654    4334 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085658    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085661    4334 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085664    4334 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085667    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085670    4334 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085673    4334 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085677    4334 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085681    4334 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085684    4334 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085687    4334 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085690    4334 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085692    4334 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085696    4334 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085700    4334 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085702    4334 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085705    4334 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085708    4334 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085711    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:42:01.095261 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085713    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085716    4334 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085719    4334 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085722    4334 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085724    4334 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085727    4334 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085730    4334 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085733    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085736    4334 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085738    4334 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085741    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085744    4334 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085746    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085749    4334 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085752    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085754    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085757    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085759    4334 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085762    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:42:01.095724 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085764    4334 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085767    4334 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085769    4334 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085772    4334 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085774    4334 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085777    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085779    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085782    4334 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085784    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085787    4334 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085789    4334 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085791    4334 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085794    4334 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085797    4334 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085799    4334 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085802    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085805    4334 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085807    4334 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085810    4334 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085813    4334 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:42:01.096228 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085816    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085819    4334 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085821    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085824    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085826    4334 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085829    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085831    4334 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085834    4334 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085836    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085839    4334 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085841    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085844    4334 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085846    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085849    4334 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085851    4334 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085854    4334 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085856    4334 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085859    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085861    4334 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:42:01.096896 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085864    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085867    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085869    4334 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085872    4334 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085874    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085877    4334 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085880    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085883    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085886    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085888    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085891    4334 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085893    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085896    4334 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085899    4334 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085901    4334 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085904    4334 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085906    4334 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085909    4334 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085911    4334 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085914    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:42:01.097694 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085917    4334 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085920    4334 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085922    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085925    4334 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085927    4334 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085930    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085934    4334 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.085938    4334 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.085966    4334 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086092    4334 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086098    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086101    4334 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086105    4334 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086107    4334 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086110    4334 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:42:01.098244 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086113    4334 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086115    4334 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086118    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086120    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086124    4334 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086126    4334 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086129    4334 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086132    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086134    4334 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086137    4334 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086139    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086142    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086144    4334 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086147    4334 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086149    4334 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086152    4334 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086154    4334 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086157    4334 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086159    4334 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086161    4334 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:42:01.098693 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086164    4334 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086166    4334 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086169    4334 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086172    4334 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086174    4334 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086176    4334 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086190    4334 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086201    4334 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086204    4334 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086207    4334 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086210    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086212    4334 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086215    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086217    4334 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086220    4334 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086223    4334 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086226    4334 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086229    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086232    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:42:01.099193 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086234    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086237    4334 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086239    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086242    4334 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086245    4334 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086247    4334 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086250    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086252    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086255    4334 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086259    4334 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086263    4334 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086266    4334 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086268    4334 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086271    4334 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086274    4334 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086276    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086279    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086281    4334 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086284    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086286    4334 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:42:01.099846 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086289    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086292    4334 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086295    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086297    4334 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086300    4334 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086302    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086305    4334 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086307    4334 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086310    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086312    4334 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086315    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086318    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086321    4334 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086323    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086326    4334 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086329    4334 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086331    4334 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086334    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086336    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:42:01.100323 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086339    4334 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086341    4334 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086344    4334 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086346    4334 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086349    4334 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086351    4334 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086354    4334 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086357    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086359    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086362    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086364    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086367    4334 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086370    4334 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086372    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086375    4334 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086378    4334 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086380    4334 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086383    4334 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086385    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086388    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:42:01.100798 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086390    4334 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086393    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.086416    4334 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086456    4334 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086460    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086464    4334 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086467    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086471    4334 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086473    4334 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086476    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086479    4334 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086482    4334 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086484    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086487    4334 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086490    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086492    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:42:01.101497 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086495    4334 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086498    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086500    4334 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086502    4334 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086505    4334 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086508    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086510    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086513    4334 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086516    4334 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086518    4334 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086521    4334 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086524    4334 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086527    4334 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086530    4334 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086533    4334 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086535    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086538    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086540    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086543    4334 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086546    4334 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:42:01.102006 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086548    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086551    4334 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086554    4334 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086557    4334 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086560    4334 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086563    4334 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086565    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086568    4334 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086570    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086574    4334 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086578    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086584    4334 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086587    4334 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086589    4334 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086592    4334 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086595    4334 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086597    4334 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086600    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086602    4334 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:42:01.102515 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086605    4334 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086608    4334 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086611    4334 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086614    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086617    4334 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086619    4334 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086622    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086625    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086627    4334 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086630    4334 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086633    4334 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086635    4334 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086638    4334 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086640    4334 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086643    4334 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086645    4334 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086648    4334 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086651    4334 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086654    4334 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086657    4334 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:42:01.103158 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086659    4334 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086662    4334 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086665    4334 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086667    4334 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086671    4334 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086674    4334 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086676    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086679    4334 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086683    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086686    4334 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086689    4334 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086691    4334 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086694    4334 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086696    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086699    4334 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086702    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086704    4334 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086707    4334 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086710    4334 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:42:01.104126 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086712    4334 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086715    4334 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086717    4334 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086720    4334 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086722    4334 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086726    4334 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086730    4334 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086733    4334 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086736    4334 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086738    4334 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086741    4334 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086744    4334 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086747    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086750    4334 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:01.086753    4334 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.086801    4334 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.087848    4334 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.087899    4334 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.088072    4334 server.go:1014] "Starting client certificate rotation"
Oct 30 02:42:01.104916 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.088220    4334 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:42:01.105590 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.088278    4334 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:36:20 +0000 UTC" deadline="2028-04-13 17:59:55.402659238 +0000 UTC"
Oct 30 02:42:01.105590 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.088304    4334 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12759h17m54.31435696s"
Oct 30 02:42:01.105590 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.088470    4334 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:42:01.105590 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.089477    4334 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:42:01.105590 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.091871    4334 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:42:01.105590 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.093404    4334 log.go:25] "Validated CRI v1 image API"
Oct 30 02:42:01.105590 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.094145    4334 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:42:01.105590 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.095654    4334 fs.go:135] Filesystem UUIDs: map[247bbc2a-230c-4710-ab71-ee258ccd3310:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ce097412-dd54-46ee-b68b-56b240f00369:/dev/nvme0n1p3]
Oct 30 02:42:01.105820 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.095675    4334 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0212d010014645cdc1d57804a89dd26b03fc3677c45a7b407b47bd8b578d709a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0212d010014645cdc1d57804a89dd26b03fc3677c45a7b407b47bd8b578d709a/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e9a5297aafc8d2b81006c842e17021e5f5bbe3490566423696e9b7d5631916c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e9a5297aafc8d2b81006c842e17021e5f5bbe3490566423696e9b7d5631916c/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/283a49d923721732fdec0697ba412765c03a3fb0f69fe586148ebfa90e3dd311/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/283a49d923721732fdec0697ba412765c03a3fb0f69fe586148ebfa90e3dd311/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30404c8c7b2776516c0bd0753098edaf124eec562767c1b5995e4af673707d83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30404c8c7b2776516c0bd0753098edaf124eec562767c1b5995e4af673707d83/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/392583615beca64131500f95cba5389c07e5ae636fb3e9e928c40e1fac6e4b29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/392583615beca64131500f95cba5389c07e5ae636fb3e9e928c40e1fac6e4b29/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43fb03925b67033cf9a96d2960dd54a9b7b358f4b165a03405c61473744b59cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43fb03925b67033cf9a96d2960dd54a9b7b358f4b165a03405c61473744b59cd/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46158511a4fd6b69bd5d6a1d126e3e938f9a98c1407c0cef949f17a93a53b875/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46158511a4fd6b69bd5d6a1d126e3e938f9a98c1407c0cef949f17a93a53b875/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86410e3fc7a266a6bd515d14c2d6e8fc36b8edef653874ba41f880dedba2c1e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86410e3fc7a266a6bd515d14c2d6e8fc36b8edef653874ba41f880dedba2c1e0/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acea34652281e42a25419e91924f50b06f6b1529982d1e4dbd016dfc6334cc78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acea34652281e42a25419e91924f50b06f6b1529982d1e4dbd016dfc6334cc78/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1d5c40431ac91b661df11beae61a7443d367df013cacb8dd5ce03f31796bd85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1d5c40431ac91b661df11beae61a7443d367df013cacb8dd5ce03f31796bd85/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b85b0e8c0d01624df34a81d113234c2a2e7e159659a33b7bb740ae92ace5148d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b85b0e8c0d01624df34a81d113234c2a2e7e159659a33b7bb740ae92ace5148d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2bd19973fbd0146c464885e597731726138d9a4cc61a2a0d651c1559a9e8e3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2bd19973fbd0146c464885e597731726138d9a4cc61a2a0d651c1559a9e8e3c/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce4a54897febdc3b7bfeb7d0455ddad06b1a91dad52d93ba1f60d6da39c06928/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce4a54897febdc3b7bfeb7d0455ddad06b1a91dad52d93ba1f60d6da39c06928/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4a0e5035706b392b895bba8ba3b3974d1a35eb8ac87cfd474be90a847b46ff5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4a0e5035706b392b895bba8ba3b3974d1a35eb8ac87cfd474be90a847b46ff5/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e42f0748a48f01ab58db0d6baf6417b098a978752d7cc8558b83078051d7c6b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e42f0748a48f01ab58db0d6baf6417b098a978752d7cc8558b83078051d7c6b1/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e62347cc65e9895ee37abee92e3e2c5ee559b1614ddce93bf56aa7c64255430d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e62347cc65e9895ee37abee92e3e2c5ee559b1614ddce93bf56aa7c64255430d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/033b7d72-7e35-47ee-9cf9-d9993d21f4be/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/033b7d72-7e35-47ee-9cf9-d9993d21f4be/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05e47302-4160-472b-988d-60585665c3bf/volumes/kubernetes.io~projected/kube-api-access-msnzw:{mountpoint:/var/lib/kubelet/pods/05e47302-4160-472b-988d-60585665c3bf/volumes/kubernetes.io~projected/kube-api-access-msnzw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1436ee1e-5bbf-488b-812b-92c7bc0906a2/volumes/kubernetes.io~projected/kube-api-access-bswjr:{mountpoint:/var/lib/kubelet/pods/1436ee1e-5bbf-488b-812b-92c7bc0906a2/volumes/kubernetes.io~projected/kube-api-access-bswjr major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1628ebb0-5634-4a42-bca2-b09ecdbfd6be/volumes/kubernetes.io~projected/kube-api-access-jhddf:{mountpoint:/var/lib/kubelet/pods/1628ebb0-5634-4a42-bca2-b09ecdbfd6be/volumes/kubernetes.io~projected/kube-api-access-jhddf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/172aafae-6ad9-420e-a213-a8bc8f2439fe/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/172aafae-6ad9-420e-a213-a8bc8f2439fe/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/180f424e-5860-452d-98b6-c0d4585dd13c/volumes/kubernetes.io~projected/kube-api-access-cfszg:{mountpoint:/var/lib/kubelet/pods/180f424e-5860-452d-98b6-c0d4585dd13c/volumes/kubernetes.io~projected/kube-api-access-cfszg major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/289d5912-6d50-4d1a-b359-1db38fd88f08/volumes/kubernetes.io~projected/kube-api-access-hqvs8:{mountpoint:/var/lib/kubelet/pods/289d5912-6d50-4d1a-b359-1db38fd88f08/volumes/kubernetes.io~projected/kube-api-access-hqvs8 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41c6efff-036c-44b7-b0ff-700a1503ec82/volumes/kubernetes.io~projected/kube-api-access-f276b:{mountpoint:/var/lib/kubelet/pods/41c6efff-036c-44b7-b0ff-700a1503ec82/volumes/kubernetes.io~projected/kube-api-access-f276b major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5264ed23-be36-4ef6-bdf7-3d694ab4ec19/volumes/kubernetes.io~projected/kube-api-access-rtqnr:{mountpoint:/var/lib/kubelet/pods/5264ed23-be36-4ef6-bdf7-3d694ab4ec19/volumes/kubernetes.io~projected/kube-api-access-rtqnr major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/818913a6-9255-4e66-b325-5ced6778ad64/volumes/kubernetes.io~projected/kube-api-access-9x24p:{mountpoint:/var/lib/kubelet/pods/818913a6-9255-4e66-b325-5ced6778ad64/volumes/kubernetes.io~projected/kube-api-access-9x24p major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b582586d-a43f-4e66-b5c2-2d83f406f0c3/volumes/kubernetes.io~projected/kube-api-access-l4sln:{mountpoint:/var/lib/kubelet/pods/b582586d-a43f-4e66-b5c2-2d83f406f0c3/volumes/kubernetes.io~projected/kube-api-access-l4sln major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b582586d-a43f-4e66-b5c2-2d83f406f0c3/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b582586d-a43f-4e66-b5c2-2d83f406f0c3/volumes/kubernetes.io~secret/signing-key major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c080eab2-cfb6-4495-ba04-475ad7344a43/volumes/kubernetes.io~projected/kube-api-access-mgmrq:{mountpoint:/var/lib/kubelet/pods/c080eab2-cfb6-4495-ba04-475ad7344a43/volumes/kubernetes.io~projected/kube-api-access-mgmrq major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c080eab2-cfb6-4495-ba04-475ad7344a43/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c080eab2-cfb6-4495-ba04-475ad7344a43/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c27c447f-f1fa-4d0e-809e-f2945b673c32/volumes/kubernetes.io~projected/kube-api-access-9tjn9:{mountpoint:/var/lib/kubelet/pods/c27c447f-f1fa-4d0e-809e-f2945b673c32/volumes/kubernetes.io~projected/kube-api-access-9tjn9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355/volumes/kubernetes.io~projected/kube-api-access-xnlwq:{mountpoint:/var/lib/kubelet/pods/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355/volumes/kubernetes.io~projected/kube-api-access-xnlwq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volumes/kubernetes.io~projected/kube-api-access-6jx4h:{mountpoint:/var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volumes/kubernetes.io~projected/kube-api-access-6jx4h major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf492558-e351-45c6-8772-2324e53b4a97/volumes/kubernetes.io~projected/kube-api-access-9mpjv:{mountpoint:/var/lib/kubelet/pods/cf492558-e351-45c6-8772-2324e53b4a97/volumes/kubernetes.io~projected/kube-api-access-9mpjv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d90d1fba-6d54-49b3-8792-36688f2242d3/volumes/kubernetes.io~projected/kube-api-access-2lgtw:{mountpoint:/var/lib/kubelet/pods/d90d1fba-6d54-49b3-8792-36688f2242d3/volumes/kubernetes.io~projected/kube-api-access-2lgtw major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~projected/kube-api-access-jmgqm:{mountpoint:/var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~projected/kube-api-access-jmgqm major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f6bb01916d14fb9e0a575efe325f7a1551e0c51d2015a079c709ab0b6c2fb213/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c88a3363b651a33090829db96d80ebbd004b4ff79681429d44b5532d9a1880dc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1e0c8d2821c8af30737d174299a2f995e03add1a3fc2705838b4aa5e9b05a117/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/68c692eb1737bc2e9a36651e59966406d77b6259c77c2fcef3701f6a6d4321df/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/23edabc0f3a632909bc84f9d9ef1d0d9196d8a27fb63f044cb11581539cb5387/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1961494401fc2dd8448d89b870f502cf7f451613c0b99327f8518e1f4e7e0032/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0b740b0936dcdbfa5156e557824f66cb639198a9091788813fa7b31b2c53b604/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/18997dee032110170a27a1eeb6d9388f5e0a8f6e6dc373dc58c2a5b0768c2bc9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/97ff7d612d17b71919e6d4a3e875a3e00a6e306f8283834aed77cd0d8a952306/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b10c1e1fcf3733a46262185ecc5da4696acdd94e38ff64b79a9af6445fe1c5df/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e1de4ecc64de891c42f1be6b4b1f852668a6d1df4bb9f86ec2767a6adac128f8/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/35e56283c172820c41401a82cf33a6e6473f82c0236a073942698dc1c1b4f257/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/e0af4e94db4504ccccd3d32ad969dfc1a086c4ec444bf78f0056802f67baf9bb/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/7b45ec22dff774fb88e3196b706f37f3dd917f096f5b3d29b1e2520c3773c9cf/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6c27e56e948882e2c1e81b7d025a770dd9b24b7ec27d1bfb542d3cbdcd6519d8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6b4eda2e05c682a3f649a99fcb353bd410121576e7e6e2c11897370f15c2aed8/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/30d4d2aa2b08d4b43a4d5942f27a9f36e3e5b2c8d6a6f85f381f37ede9120c09/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5b71e6abc905390ba365074365b98fbe76a2ee8c4ce2c04505fdacaabf1fd490/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/97dcf0e58c9b5e77d6362faa51fb442213529bfa2d26b242091c7ee7f7ac0ab2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1a59b7078dd82f58b9fea44e6327ace64c5be9bb38f088a935bb21e754f81e5c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/fd06824434a1ffe987b5d19d6ec738c154f1ef2a3a9888310ef116d4077abc46/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/c8ebb4443ee4ab87ce8f2e83833f39a776d877248f984b1b7e6c7af6ffbb9751/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/f2c6ba2728de5b0e75cb72dfadc4f3d1667971edd197e2215bebd9de26db3764/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/2f01b31ec5da9896bd4642433e1fb87ab3059ab16765ac8440df56af26aad19e/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/8d53af67d9fb8df0d5e384c3b09c7117ecafa277a609337810b34ab79d8918ad/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/6a339363ee89a33709293b30f620674e739cbfaba3cabfc2969d3c258ac07986/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/f309919b905e78b8181c620efe856a7a1d462900e27dc8f499a99ea5ed80e524/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/42b6e6019f399dcc9d8fb55c4e051fe67a65343490fa3c21cdcb6cafc28f8a4f/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/677fa6c5f9744171ca4b29bdd43de7404d08fa8793029d6266a3307f3dd1792c/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/42a2416a554049d85011df6e677b0352aa07d0d66c3a3c279c99988b5ab4cf03/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/dc184d27eb786e7ccc5304050f6f5006982e3695cef30f2ab1b50b6db448c9b1/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/7d8e3df6f6a13dd10438341473917c3ac1acd66b9deac660e9eb5a739cdff85a/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/b62aecbdf6ad083d10e2eff1d3f60aefc844250e546282519a0f61fef24664d9/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/4157e234380e67eafbc592f41f5fde7b5d49006b2e70afc687a5143ddeb974fb/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a1beba8d74ca5be514c5b84e06e2615d249c04640badd84e6415b8e26e264a86/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ba9446181b56302d79005702598ad04293f93991d9e17e71b15be2f889011e4b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/fa327240f4128307f88379ba4da4e00ad20d345dff07b2940555cd87fa23065b/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f4f7640f112ae6ec57f7525ce060deb17c5a29e0cdab01a638767cf78ee1b1af/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/282e5f843fcc3a0428028b599d976e56d2325ef440ab31ff40deaa02f9667687/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/ca94d651e8cea3e90646309e8d2c89f39891a56c0c917afe5a23f714512b83fb/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ae8f45a746e0bf6325da3b8f03c70f3067f025db8af03f631a97074f52648ca3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/756b14b82be82415aee3fc1ee35c2fb13b5d4a2e52ed372fb5a3f4c166567f77/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/9da93202a6da202a9681979bab1ed9f2e44646dd9e995437232bf0203d38c1a6/merged major:0 minor:94 fsType:overlay blockSize:0}]
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.102332    4334 manager.go:217] Machine: {Timestamp:2026-10-30 02:42:01.101843252 +0000 UTC m=+0.108433496 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100037 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9636410522fa34c0a2cf3ad88943 SystemUUID:ec2a9636-4105-22fa-34c0-a2cf3ad88943 BootID:5e824f4c-6520-45b2-963d-8ba7d110686f Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b85b0e8c0d01624df34a81d113234c2a2e7e159659a33b7bb740ae92ace5148d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0212d010014645cdc1d57804a89dd26b03fc3677c45a7b407b47bd8b578d709a/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1436ee1e-5bbf-488b-812b-92c7bc0906a2/volumes/kubernetes.io~projected/kube-api-access-bswjr DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/180f424e-5860-452d-98b6-c0d4585dd13c/volumes/kubernetes.io~projected/kube-api-access-cfszg DeviceMajor:0 DeviceMinor:216 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acea34652281e42a25419e91924f50b06f6b1529982d1e4dbd016dfc6334cc78/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c080eab2-cfb6-4495-ba04-475ad7344a43/volumes/kubernetes.io~projected/kube-api-access-mgmrq DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43fb03925b67033cf9a96d2960dd54a9b7b358f4b165a03405c61473744b59cd/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b582586d-a43f-4e66-b5c2-2d83f406f0c3/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:231 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e62347cc65e9895ee37abee92e3e2c5ee559b1614ddce93bf56aa7c64255430d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4a0e5035706b392b895bba8ba3b3974d1a35eb8ac87cfd474be90a847b46ff5/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/05e47302-4160-472b-988d-60585665c3bf/volumes/kubernetes.io~projected/kube-api-access-msnzw DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41c6efff-036c-44b7-b0ff-700a1503ec82/volumes/kubernetes.io~projected/kube-api-access-f276b DeviceMajor:0 DeviceMinor:215 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/289d5912-6d50-4d1a-b359-1db38fd88f08/volumes/kubernetes.io~projected/kube-api-access-hqvs8 DeviceMajor:0 DeviceMinor:228 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce4a54897febdc3b7bfeb7d0455ddad06b1a91dad52d93ba1f60d6da39c06928/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86410e3fc7a266a6bd515d14c2d6e8fc36b8edef653874ba41f880dedba2c1e0/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/033b7d72-7e35-47ee-9cf9-d9993d21f4be/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5264ed23-be36-4ef6-bdf7-3d694ab4ec19/volumes/kubernetes.io~projected/kube-api-access-rtqnr DeviceMajor:0 DeviceMinor:217 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cf492558-e351-45c6-8772-2324e53b4a97/volumes/kubernetes.io~projected/kube-api-access-9mpjv DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30404c8c7b2776516c0bd0753098edaf124eec562767c1b5995e4af673707d83/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/818913a6-9255-4e66-b325-5ced6778ad64/volumes/kubernetes.io~projected/kube-api-access-9x24p DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355/volumes/kubernetes.io~projected/kube-api-access-xnlwq DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c27c447f-f1fa-4d0e-809e-f2945b673c32/volumes/kubernetes.io~projected/kube-api-access-9tjn9 DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/283a49d923721732fdec0697ba412765c03a3fb0f69fe586148ebfa90e3dd311/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volumes/kubernetes.io~projected/kube-api-access-6jx4h DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46158511a4fd6b69bd5d6a1d126e3e938f9a98c1407c0cef949f17a93a53b875/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e42f0748a48f01ab58db0d6baf6417b098a978752d7cc8558b83078051d7c6b1/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b582586d-a43f-4e66-b5c2-2d83f406f0c3/volumes/kubernetes.io~projected/kube-api-access-l4sln DeviceMajor:0 DeviceMinor:232 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e9a5297aafc8d2b81006c842e17021e5f5bbe3490566423696e9b7d5631916c/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cea2f02d-9272-44dc-acd9-d899a2d711a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/172aafae-6ad9-420e-a213-a8bc8f2439fe/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d90d1fba-6d54-49b3-8792-36688f2242d3/volumes/kubernetes.io~projected/kube-api-access-2lgtw DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/392583615beca64131500f95cba5389c07e5ae636fb3e9e928c40e1fac6e4b29/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1d5c40431ac91b661df11beae61a7443d367df013cacb8dd5ce03f31796bd85/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2bd19973fbd0146c464885e597731726138d9a4cc61a2a0d651c1559a9e8e3c/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb013b42-f40c-4df9-bd10-ccb8975e54dc/volumes/kubernetes.io~projected/kube-api-access-jmgqm DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1628ebb0-5634-4a42-bca2-b09ecdbfd6be/volumes/kubernetes.io~projected/kube-api-access-jhddf DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c080eab2-cfb6-4495-ba04-475ad7344a43/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:283a49d92372173 MacAddress:d6:17:28:b0:fa:6a Speed:10000 Mtu:8901} {Name:30404c8c7b27765 MacAddress:a2:22:2e:93:bf:c2 Speed:10000 Mtu:8901} {Name:43fb03925b67033 MacAddress:26:1f:ef:25:be:38 Speed:10000 Mtu:8901} {Name:acea34652281e42 MacAddress:52:64:02:20:09:ba Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e4:b9:a8:ff Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:11:ec:c5:60:be Speed:0 Mtu:8901} {Name:c2bd19973fbd014 MacAddress:ce:45:5d:04:a7:12 Speed:10000 Mtu:8901} {Name:ce4a54897febdc3 MacAddress:a2:94:d6:7b:6b:c2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e4:b9:a8:ff Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3e:01:97:61:21:15 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:88:a5:df:32:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.102609    4334 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.102682    4334 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103056    4334 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103081    4334 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103261    4334 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103274    4334 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103287    4334 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103313    4334 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103408    4334 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103642    4334 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.103998    4334 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.104044    4334 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.104063    4334 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.104076    4334 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.104088    4334 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.104100    4334 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.104823    4334 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.104836    4334 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:42:01.105978 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.105313    4334 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:42:01.107165 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107152    4334 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:42:01.107165 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107165    4334 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:42:01.107265 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107255    4334 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:42:01.107432 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107422    4334 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:42:01.107485 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107468    4334 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:42:01.107563 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107554    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:42:01.107598 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107568    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:42:01.107598 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107574    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:42:01.107598 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107580    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:42:01.107598 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107585    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:42:01.107598 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107591    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:42:01.107598 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107597    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:42:01.107772 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107602    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:42:01.107772 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107609    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:42:01.107772 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107614    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:42:01.107772 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107628    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:42:01.107772 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107639    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:42:01.107772 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107663    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:42:01.107772 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.107668    4334 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:42:01.110887 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.110872    4334 server.go:1263] "Started kubelet"
Oct 30 02:42:01.110993 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.110944    4334 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:42:01.111682 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.111109    4334 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:42:01.111682 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.111196    4334 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:42:01.112479 ip-10-0-133-197 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:42:01.113996 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.113571    4334 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:42:01.117412 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.117392    4334 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:42:01.117809 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.117790    4334 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:42:01.118238 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.118111    4334 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:42:01.118316 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.118280    4334 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:36:22 +0000 UTC" deadline="2028-05-22 09:12:09.439987726 +0000 UTC"
Oct 30 02:42:01.118316 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.118298    4334 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13686h30m8.321692564s"
Oct 30 02:42:01.118467 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.118440    4334 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:42:01.118467 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.118467    4334 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:42:01.118584 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.118553    4334 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:42:01.119694 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119666    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="180f424e-5860-452d-98b6-c0d4585dd13c" volumeName="kubernetes.io/empty-dir/180f424e-5860-452d-98b6-c0d4585dd13c-data-volume" seLinuxMountContext=""
Oct 30 02:42:01.119694 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119693    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" volumeName="kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119706    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea2f02d-9272-44dc-acd9-d899a2d711a9" volumeName="kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119718    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e47302-4160-472b-988d-60585665c3bf" volumeName="kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119728    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1628ebb0-5634-4a42-bca2-b09ecdbfd6be" volumeName="kubernetes.io/configmap/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-serviceca" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119740    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" volumeName="kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119751    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b582586d-a43f-4e66-b5c2-2d83f406f0c3" volumeName="kubernetes.io/secret/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-key" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119761    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e47302-4160-472b-988d-60585665c3bf" volumeName="kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119774    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1436ee1e-5bbf-488b-812b-92c7bc0906a2" volumeName="kubernetes.io/configmap/1436ee1e-5bbf-488b-812b-92c7bc0906a2-config-volume" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119784    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1436ee1e-5bbf-488b-812b-92c7bc0906a2" volumeName="kubernetes.io/projected/1436ee1e-5bbf-488b-812b-92c7bc0906a2-kube-api-access-bswjr" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119797    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea2f02d-9272-44dc-acd9-d899a2d711a9" volumeName="kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-env-overrides" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119810    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1436ee1e-5bbf-488b-812b-92c7bc0906a2" volumeName="kubernetes.io/empty-dir/1436ee1e-5bbf-488b-812b-92c7bc0906a2-tmp-dir" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119822    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" volumeName="kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path" seLinuxMountContext=""
Oct 30 02:42:01.119839 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119833    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="818913a6-9255-4e66-b325-5ced6778ad64" volumeName="kubernetes.io/empty-dir/818913a6-9255-4e66-b325-5ced6778ad64-tmp-dir" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119847    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c27c447f-f1fa-4d0e-809e-f2945b673c32" volumeName="kubernetes.io/projected/c27c447f-f1fa-4d0e-809e-f2945b673c32-kube-api-access-9tjn9" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119858    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea2f02d-9272-44dc-acd9-d899a2d711a9" volumeName="kubernetes.io/projected/cea2f02d-9272-44dc-acd9-d899a2d711a9-kube-api-access-6jx4h" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119870    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90d1fba-6d54-49b3-8792-36688f2242d3" volumeName="kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119881    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90d1fba-6d54-49b3-8792-36688f2242d3" volumeName="kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119894    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb013b42-f40c-4df9-bd10-ccb8975e54dc" volumeName="kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-tmp" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119905    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="172aafae-6ad9-420e-a213-a8bc8f2439fe" volumeName="kubernetes.io/secret/172aafae-6ad9-420e-a213-a8bc8f2439fe-agent-certs" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119917    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="180f424e-5860-452d-98b6-c0d4585dd13c" volumeName="kubernetes.io/projected/180f424e-5860-452d-98b6-c0d4585dd13c-kube-api-access-cfszg" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119928    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b582586d-a43f-4e66-b5c2-2d83f406f0c3" volumeName="kubernetes.io/projected/b582586d-a43f-4e66-b5c2-2d83f406f0c3-kube-api-access-l4sln" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119940    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d43c51-c7ff-4751-9c3b-0eeae8d6a355" volumeName="kubernetes.io/configmap/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119953    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d43c51-c7ff-4751-9c3b-0eeae8d6a355" volumeName="kubernetes.io/projected/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-kube-api-access-xnlwq" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119979    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea2f02d-9272-44dc-acd9-d899a2d711a9" volumeName="kubernetes.io/secret/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.119990    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb013b42-f40c-4df9-bd10-ccb8975e54dc" volumeName="kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-tuned" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120002    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="033b7d72-7e35-47ee-9cf9-d9993d21f4be" volumeName="kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120014    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41c6efff-036c-44b7-b0ff-700a1503ec82" volumeName="kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120030    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="818913a6-9255-4e66-b325-5ced6778ad64" volumeName="kubernetes.io/projected/818913a6-9255-4e66-b325-5ced6778ad64-kube-api-access-9x24p" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120042    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea2f02d-9272-44dc-acd9-d899a2d711a9" volumeName="kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-config" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120053    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90d1fba-6d54-49b3-8792-36688f2242d3" volumeName="kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120066    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90d1fba-6d54-49b3-8792-36688f2242d3" volumeName="kubernetes.io/projected/d90d1fba-6d54-49b3-8792-36688f2242d3-kube-api-access-2lgtw" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120090    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb013b42-f40c-4df9-bd10-ccb8975e54dc" volumeName="kubernetes.io/projected/fb013b42-f40c-4df9-bd10-ccb8975e54dc-kube-api-access-jmgqm" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120102    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e47302-4160-472b-988d-60585665c3bf" volumeName="kubernetes.io/projected/05e47302-4160-472b-988d-60585665c3bf-kube-api-access-msnzw" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120114    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="172aafae-6ad9-420e-a213-a8bc8f2439fe" volumeName="kubernetes.io/configmap/172aafae-6ad9-420e-a213-a8bc8f2439fe-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120125    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="289d5912-6d50-4d1a-b359-1db38fd88f08" volumeName="kubernetes.io/projected/289d5912-6d50-4d1a-b359-1db38fd88f08-kube-api-access-hqvs8" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120136    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b582586d-a43f-4e66-b5c2-2d83f406f0c3" volumeName="kubernetes.io/configmap/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-cabundle" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120149    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c080eab2-cfb6-4495-ba04-475ad7344a43" volumeName="kubernetes.io/projected/c080eab2-cfb6-4495-ba04-475ad7344a43-kube-api-access-mgmrq" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120161    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" volumeName="kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120172    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c080eab2-cfb6-4495-ba04-475ad7344a43" volumeName="kubernetes.io/secret/c080eab2-cfb6-4495-ba04-475ad7344a43-cert" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120184    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf492558-e351-45c6-8772-2324e53b4a97" volumeName="kubernetes.io/projected/cf492558-e351-45c6-8772-2324e53b4a97-kube-api-access-9mpjv" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120197    4334 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1628ebb0-5634-4a42-bca2-b09ecdbfd6be" volumeName="kubernetes.io/projected/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-kube-api-access-jhddf" seLinuxMountContext=""
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120210    4334 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:42:01.120379 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.120218    4334 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:42:01.121845 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.121415    4334 factory.go:55] Registering systemd factory
Oct 30 02:42:01.121845 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.121431    4334 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:42:01.121953 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:01.121828    4334 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.ec2.internal\" not found"
Oct 30 02:42:01.122785 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.122766    4334 factory.go:153] Registering CRI-O factory
Oct 30 02:42:01.122785 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.122782    4334 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:42:01.122944 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.122832    4334 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:42:01.122944 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.122852    4334 factory.go:103] Registering Raw factory
Oct 30 02:42:01.122944 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.122864    4334 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:42:01.123123 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.123107    4334 manager.go:319] Starting recovery of all containers
Oct 30 02:42:01.151320 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.151281    4334 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:42:01.152601 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.152580    4334 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:42:01.152601 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.152600    4334 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:42:01.152751 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.152624    4334 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:42:01.152751 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:01.152664    4334 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:42:01.162496 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.162472    4334 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:42:01.162664 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.162643    4334 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:42:01.164890 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.164864    4334 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="4d29b2aba50ff427b30eee7815c84e07dd24265ec6cc436ac9fc2eb83426d516" exitCode=0
Oct 30 02:42:01.164890 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.164888    4334 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="f640f3d45bc40a7c7915c77913e520d77dc23b6d38d21459a12c5f8f56b652cf" exitCode=0
Oct 30 02:42:01.165000 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.164896    4334 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="dcad8297433ae34a8bf36551e1c0f147b2e1170c8f3869dcf108acc2b37d4a8d" exitCode=0
Oct 30 02:42:01.165000 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.164905    4334 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="1d7e68d93539de3709e0d3d64b26f0fdc079e8e1d2e12b781da68dad2dac51ef" exitCode=0
Oct 30 02:42:01.165000 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.164915    4334 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="5ff83304ce1d5bfc11112bf5dd255409c5b37a8d9bfb121dc053765be70950a6" exitCode=0
Oct 30 02:42:01.165000 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.164922    4334 generic.go:356] "Generic (PLEG): container finished" podID="d90d1fba-6d54-49b3-8792-36688f2242d3" containerID="90333823464faa361ab4edff2aace20a9d9e02e638942e7e6518900b08a47a8b" exitCode=0
Oct 30 02:42:01.174284 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.174221    4334 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:42:01.175864 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.175841    4334 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:42:01.175864 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.175852    4334 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:42:01.190823 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.190798    4334 generic.go:356] "Generic (PLEG): container finished" podID="a68cf21ce8d2089c87349d6b232245b0" containerID="62fcf0cd130d648fe8d4053f0e005f5a362504d9f030638a1e04787af6e3ae79" exitCode=0
Oct 30 02:42:01.199070 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.199055    4334 manager.go:324] Recovery completed
Oct 30 02:42:01.214194 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214176    4334 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:42:01.214194 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214190    4334 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:42:01.214304 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214206    4334 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:42:01.214362 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214349    4334 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:42:01.214395 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214362    4334 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:42:01.214395 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214379    4334 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:42:01.214395 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214384    4334 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:42:01.214395 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214390    4334 policy_none.go:50] "Start"
Oct 30 02:42:01.214566 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214398    4334 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:42:01.214566 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.214407    4334 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:42:01.220656 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.220638    4334 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:42:01.220656 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.220652    4334 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:42:01.220656 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.220658    4334 policy_none.go:44] "Start"
Oct 30 02:42:01.225359 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.225330    4334 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:42:01.229505 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.229491    4334 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:42:01.230152 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:01.229514    4334 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:42:01.230202 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.230158    4334 server.go:87] "Starting device plugin registration server"
Oct 30 02:42:01.230479 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.230466    4334 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:42:01.230525 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.230484    4334 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:42:01.230591 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.230572    4334 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:42:01.230705 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.230666    4334 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:42:01.230705 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.230709    4334 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:42:01.230859 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.230716    4334 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:42:01.230859 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.230777    4334 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:42:01.230699338Z","UUID":"55cf7312-9647-46a4-a5a1-4d912333293b","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:42:01.232052 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:01.232030    4334 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:42:01.232136 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.232075    4334 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:42:01.232136 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.232110    4334 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:42:01.252749 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.252712    4334 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"]
Oct 30 02:42:01.253307 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.253293    4334 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.253362 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.253293    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal" event={"ID":"adf118574a1cbe5f944e98682e2ff56b","Type":"ContainerStarted","Data":"4f950047b0f58b56268ceb2299d993c3c9f862b001011ae12f57e0851cda4ab1"}
Oct 30 02:42:01.253362 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.253335    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal" event={"ID":"adf118574a1cbe5f944e98682e2ff56b","Type":"ContainerStarted","Data":"d4a0e5035706b392b895bba8ba3b3974d1a35eb8ac87cfd474be90a847b46ff5"}
Oct 30 02:42:01.253473 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.253373    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal" event={"ID":"a68cf21ce8d2089c87349d6b232245b0","Type":"ContainerStarted","Data":"e5daf254c131b715ed7f719498977748c60428853db4e4d135882375c52fb328"}
Oct 30 02:42:01.253473 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.253391    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal" event={"ID":"a68cf21ce8d2089c87349d6b232245b0","Type":"ContainerDied","Data":"62fcf0cd130d648fe8d4053f0e005f5a362504d9f030638a1e04787af6e3ae79"}
Oct 30 02:42:01.253473 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.253409    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal" event={"ID":"a68cf21ce8d2089c87349d6b232245b0","Type":"ContainerStarted","Data":"b85b0e8c0d01624df34a81d113234c2a2e7e159659a33b7bb740ae92ace5148d"}
Oct 30 02:42:01.253583 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.253552    4334 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.275922 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.275901    4334 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:42:01.276005 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:01.275941    4334 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-197.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.276993 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.276973    4334 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:42:01.277064 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:01.277038    4334 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.330941 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.330916    4334 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:42:01.332175 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.332160    4334 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:42:01.332248 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.332186    4334 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:42:01.332248 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.332198    4334 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:42:01.332248 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.332231    4334 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.363584 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.363561    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf118574a1cbe5f944e98682e2ff56b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.ec2.internal\" (UID: \"adf118574a1cbe5f944e98682e2ff56b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.363584 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.363586    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.363719 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.363604    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.364781 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.364763    4334 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.364828 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.364824    4334 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.464399 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.464339    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf118574a1cbe5f944e98682e2ff56b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.ec2.internal\" (UID: \"adf118574a1cbe5f944e98682e2ff56b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.464399 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.464368    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.464399 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.464389    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.464649 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.464442    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.464649 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.464483    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf118574a1cbe5f944e98682e2ff56b-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.ec2.internal\" (UID: \"adf118574a1cbe5f944e98682e2ff56b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:01.464649 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:01.464507    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a68cf21ce8d2089c87349d6b232245b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" (UID: \"a68cf21ce8d2089c87349d6b232245b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:02.105226 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.105173    4334 apiserver.go:52] "Watching apiserver"
Oct 30 02:42:02.158725 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.158697    4334 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:42:02.159121 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.159101    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal","kube-system/konnectivity-agent-mkxt8","openshift-insights/periodic-gathering-jbbb7-m2nlh","openshift-multus/multus-additional-cni-plugins-hp7gt","openshift-ovn-kubernetes/ovnkube-node-zdxqj","openshift-dns/node-resolver-k5blc","openshift-image-registry/node-ca-qb86g","openshift-dns/dns-default-8k7nc","openshift-network-operator/iptables-alerter-8l6sk","kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vj6lt","openshift-service-ca/service-ca-5fffc97bdf-xhrrk","openshift-ingress-canary/ingress-canary-5tmj8","openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx","openshift-network-diagnostics/network-check-target-v7c87","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg","openshift-insights/insights-runtime-extractor-bvj8c","openshift-multus/multus-pvqdb","openshift-multus/network-metrics-daemon-gztvh","kube-system/global-pull-secret-syncer-6xh6w"]
Oct 30 02:42:02.161667 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.161647    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:02.163834 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.163818    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.167434 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.167376    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.176219 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.176202    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:42:02.176422 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.176408    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.176508 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.176443    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.176580 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.176544    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.182277 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.182257    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vzqmp\""
Oct 30 02:42:02.182594 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.182578    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d89pf\""
Oct 30 02:42:02.182736 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.182724    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:42:02.182780 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.182749    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:42:02.183064 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.183052    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-stwqd\""
Oct 30 02:42:02.193506 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.193490    4334 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:02.193623 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.193610    4334 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:02.210807 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.210786    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:42:02.210971 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.210952    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.210971 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.210958    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.211106 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.210996    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.211106 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211038    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:42:02.211106 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211062    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.211106 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211079    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.211106 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211094    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:42:02.211330 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211110    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.211330 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211170    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.211330 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211197    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.211655 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211631    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:42:02.211723 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211660    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:42:02.211723 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211681    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 02:42:02.211723 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211712    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.211875 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211720    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.211875 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211772    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.211875 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211795    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:42:02.211875 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211812    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.211875 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211824    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:42:02.211875 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211837    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:42:02.211875 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211846    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211895    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211951    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.211961    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212067    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212086    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212115    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212143    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212173    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:42:02.212184 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212185    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:42:02.212608 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212187    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:42:02.212608 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212189    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:42:02.212608 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212208    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:42:02.212608 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.212304    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:42:02.220368 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.220353    4334 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:42:02.238197 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.238178    4334 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:42:02.238286 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.238227    4334 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:02.241110 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.241094    4334 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:42:02.241194 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.241127    4334 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-197.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.ec2.internal"
Oct 30 02:42:02.264020 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.264002    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qb86g_1628ebb0-5634-4a42-bca2-b09ecdbfd6be/node-ca/0.log"
Oct 30 02:42:02.268198 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268181    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhddf\" (UniqueName: \"kubernetes.io/projected/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-kube-api-access-jhddf\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:42:02.268259 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268208    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-netns\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.268259 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268224    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswjr\" (UniqueName: \"kubernetes.io/projected/1436ee1e-5bbf-488b-812b-92c7bc0906a2-kube-api-access-bswjr\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.268259 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268239    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-cnibin\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.268343 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268257    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.268343 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268292    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-system-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.268343 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268323    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-etc-kubernetes\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.268489 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268442    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/172aafae-6ad9-420e-a213-a8bc8f2439fe-konnectivity-ca\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:42:02.268530 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268510    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-kubelet-config\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:42:02.268579 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268540    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-netd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.268624 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268574    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpjv\" (UniqueName: \"kubernetes.io/projected/cf492558-e351-45c6-8772-2324e53b4a97-kube-api-access-9mpjv\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:02.268624 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268614    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-run\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.268694 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268673    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-var-lib-kubelet\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.268736 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268717    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/172aafae-6ad9-420e-a213-a8bc8f2439fe-konnectivity-ca\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:42:02.268774 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268758    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-bin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.268811 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268785    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:42:02.268854 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268829    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-host\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:42:02.268854 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268850    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-node-log\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.268919 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268881    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-log-socket\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.268919 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.268905    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-tuned\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.269029 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269009    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-tuned\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.269081 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269053    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180f424e-5860-452d-98b6-c0d4585dd13c-crio-socket\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.269131 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269112    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.269168 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269147    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-socket-dir-parent\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.269268 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269172    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-kubelet\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.269268 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269208    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:02.269268 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269253    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-kube-api-access-xnlwq\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:42:02.269406 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269289    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.269406 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269318    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-hostroot\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.269406 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269370    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.269574 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269411    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-script-lib\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.269574 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269467    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-tmp\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.269574 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269537    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-multus-daemon-config\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.269574 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269550    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb013b42-f40c-4df9-bd10-ccb8975e54dc-tmp\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.269574 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269553    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.269788 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269588    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:42:02.269788 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269619    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-dbus\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:42:02.269788 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269645    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-slash\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.269788 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269688    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1436ee1e-5bbf-488b-812b-92c7bc0906a2-config-volume\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.269788 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269755    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1436ee1e-5bbf-488b-812b-92c7bc0906a2-tmp-dir\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.269788 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269773    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmgqm\" (UniqueName: \"kubernetes.io/projected/fb013b42-f40c-4df9-bd10-ccb8975e54dc-kube-api-access-jmgqm\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.269788 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269779    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-multus-daemon-config\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269782    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-script-lib\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269831    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269839    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269852    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1436ee1e-5bbf-488b-812b-92c7bc0906a2-tmp-dir\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269872    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-serviceca\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269890    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-registration-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269950    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.269993    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:42:02.270054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270009    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1436ee1e-5bbf-488b-812b-92c7bc0906a2-config-volume\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.270323 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270055    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-serviceca\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:42:02.270323 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270070    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.270323 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270111    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfszg\" (UniqueName: \"kubernetes.io/projected/180f424e-5860-452d-98b6-c0d4585dd13c-kube-api-access-cfszg\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.270323 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270127    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:42:02.270323 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270159    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.270323 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270272    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.270527 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270330    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-device-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.270527 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270359    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-sys-fs\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.270527 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270404    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-systemd-units\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.270527 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270495    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-bin\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.270652 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270534    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.270652 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270560    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-key\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:42:02.270652 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270577    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.270652 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270603    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-systemd\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.270840 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270702    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvs8\" (UniqueName: \"kubernetes.io/projected/289d5912-6d50-4d1a-b359-1db38fd88f08-kube-api-access-hqvs8\") pod \"migrator-66df5d496b-z89rx\" (UID: \"289d5912-6d50-4d1a-b359-1db38fd88f08\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"
Oct 30 02:42:02.270840 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270757    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-socket-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.270840 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270796    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.270928 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270835    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-k8s-cni-cncf-io\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.270928 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270876    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-multus\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.270993 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270952    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:42:02.270993 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270959    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sln\" (UniqueName: \"kubernetes.io/projected/b582586d-a43f-4e66-b5c2-2d83f406f0c3-kube-api-access-l4sln\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:42:02.271046 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.270998    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x24p\" (UniqueName: \"kubernetes.io/projected/818913a6-9255-4e66-b325-5ced6778ad64-kube-api-access-9x24p\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:42:02.271046 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271023    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjn9\" (UniqueName: \"kubernetes.io/projected/c27c447f-f1fa-4d0e-809e-f2945b673c32-kube-api-access-9tjn9\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.271113 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271049    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgmrq\" (UniqueName: \"kubernetes.io/projected/c080eab2-cfb6-4495-ba04-475ad7344a43-kube-api-access-mgmrq\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:42:02.271174 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271159    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-systemd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.271206 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271183    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-etc-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.271239 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271221    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-ovn\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.271268 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271240    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-kubelet\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.271268 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271256    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.271330 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271312    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c080eab2-cfb6-4495-ba04-475ad7344a43-cert\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:42:02.271382 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271353    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-var-lib-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.271429 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271400    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:42:02.271487 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271472    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-modprobe-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.271519 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271496    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-host\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.271548 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271520    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.271576 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271569    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.271612 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271588    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysconfig\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.271645 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271609    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180f424e-5860-452d-98b6-c0d4585dd13c-data-volume\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.271645 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271631    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-cni-binary-copy\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.271716 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271682    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/180f424e-5860-452d-98b6-c0d4585dd13c-data-volume\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.271716 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271705    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-netns\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.271787 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271720    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-multus-certs\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.271787 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271730    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:42:02.271787 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271739    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818913a6-9255-4e66-b325-5ced6778ad64-tmp-dir\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:42:02.271907 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271839    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05e47302-4160-472b-988d-60585665c3bf-cni-binary-copy\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.271907 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271873    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:42:02.271997 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271897    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818913a6-9255-4e66-b325-5ced6778ad64-tmp-dir\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:42:02.271997 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271911    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-env-overrides\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.271997 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271910    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d90d1fba-6d54-49b3-8792-36688f2242d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.271997 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.271944    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-kubernetes\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.272105 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272022    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-sys\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.272105 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272066    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-lib-modules\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.272105 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272089    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lgtw\" (UniqueName: \"kubernetes.io/projected/d90d1fba-6d54-49b3-8792-36688f2242d3-kube-api-access-2lgtw\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.272105 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272076    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-env-overrides\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.272213 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272114    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/033b7d72-7e35-47ee-9cf9-d9993d21f4be-original-pull-secret\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:42:02.272213 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272124    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-config\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.272213 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272147    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovn-node-metrics-cert\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.272213 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272194    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-conf\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.272330 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272222    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-cnibin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.272330 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272305    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-conf-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.272390 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272331    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msnzw\" (UniqueName: \"kubernetes.io/projected/05e47302-4160-472b-988d-60585665c3bf-kube-api-access-msnzw\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.272390 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272346    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818913a6-9255-4e66-b325-5ced6778ad64-hosts-file\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:42:02.272390 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272378    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovnkube-config\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.272495 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272412    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqnr\" (UniqueName: \"kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:42:02.272495 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272442    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.272591 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272576    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jx4h\" (UniqueName: \"kubernetes.io/projected/cea2f02d-9272-44dc-acd9-d899a2d711a9-kube-api-access-6jx4h\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.272624 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272612    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-iptables-alerter-script\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:42:02.272680 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272662    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-host-slash\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:42:02.272711 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272691    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-system-cni-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.272743 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272714    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/172aafae-6ad9-420e-a213-a8bc8f2439fe-agent-certs\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:42:02.272887 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272874    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-iptables-alerter-script\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:42:02.273010 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.272997    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/172aafae-6ad9-420e-a213-a8bc8f2439fe-agent-certs\") pod \"konnectivity-agent-mkxt8\" (UID: \"172aafae-6ad9-420e-a213-a8bc8f2439fe\") " pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:42:02.277622 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.277606    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:42:02.277749 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.277735    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:42:02.277815 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.277738    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:42:02.277871 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.277858    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zm8vf\""
Oct 30 02:42:02.277995 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.277974    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-29rqz\""
Oct 30 02:42:02.278131 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.278113    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:42:02.278240 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.278223    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hjgzr\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.278518    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.278544    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.278569    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6bd55\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.278698    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-blzpj\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.278757    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.278949    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j74dx\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279045    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7d7z9\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279141    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcz9c\""
Oct 30 02:42:02.279163 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279159    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8z95j\""
Oct 30 02:42:02.279464 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279217    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hbwgl\""
Oct 30 02:42:02.279464 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279417    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zz8vj\""
Oct 30 02:42:02.279464 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279420    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tbftn\""
Oct 30 02:42:02.279613 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279596    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-czntg\""
Oct 30 02:42:02.279679 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279642    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kxfcp\""
Oct 30 02:42:02.279679 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.279647    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2wtbz\""
Oct 30 02:42:02.281753 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.281733    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b582586d-a43f-4e66-b5c2-2d83f406f0c3-signing-key\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:42:02.281799 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.281776    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c080eab2-cfb6-4495-ba04-475ad7344a43-cert\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:42:02.282810 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.282787    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cea2f02d-9272-44dc-acd9-d899a2d711a9-ovn-node-metrics-cert\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.288900 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.288883    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpjv\" (UniqueName: \"kubernetes.io/projected/cf492558-e351-45c6-8772-2324e53b4a97-kube-api-access-9mpjv\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:02.289545 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.289528    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmgqm\" (UniqueName: \"kubernetes.io/projected/fb013b42-f40c-4df9-bd10-ccb8975e54dc-kube-api-access-jmgqm\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.289760 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.289745    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgmrq\" (UniqueName: \"kubernetes.io/projected/c080eab2-cfb6-4495-ba04-475ad7344a43-kube-api-access-mgmrq\") pod \"ingress-canary-5tmj8\" (UID: \"c080eab2-cfb6-4495-ba04-475ad7344a43\") " pod="openshift-ingress-canary/ingress-canary-5tmj8"
Oct 30 02:42:02.292228 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292089    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhddf\" (UniqueName: \"kubernetes.io/projected/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-kube-api-access-jhddf\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:42:02.292576 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292540    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x24p\" (UniqueName: \"kubernetes.io/projected/818913a6-9255-4e66-b325-5ced6778ad64-kube-api-access-9x24p\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:42:02.292697 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292589    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msnzw\" (UniqueName: \"kubernetes.io/projected/05e47302-4160-472b-988d-60585665c3bf-kube-api-access-msnzw\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.292697 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292614    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfszg\" (UniqueName: \"kubernetes.io/projected/180f424e-5860-452d-98b6-c0d4585dd13c-kube-api-access-cfszg\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.292697 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292639    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lgtw\" (UniqueName: \"kubernetes.io/projected/d90d1fba-6d54-49b3-8792-36688f2242d3-kube-api-access-2lgtw\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.292697 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292643    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvs8\" (UniqueName: \"kubernetes.io/projected/289d5912-6d50-4d1a-b359-1db38fd88f08-kube-api-access-hqvs8\") pod \"migrator-66df5d496b-z89rx\" (UID: \"289d5912-6d50-4d1a-b359-1db38fd88f08\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z89rx"
Oct 30 02:42:02.292934 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292860    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f276b\" (UniqueName: \"kubernetes.io/projected/41c6efff-036c-44b7-b0ff-700a1503ec82-kube-api-access-f276b\") pod \"network-check-target-v7c87\" (UID: \"41c6efff-036c-44b7-b0ff-700a1503ec82\") " pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:42:02.292997 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292949    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswjr\" (UniqueName: \"kubernetes.io/projected/1436ee1e-5bbf-488b-812b-92c7bc0906a2-kube-api-access-bswjr\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.292997 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.292973    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjn9\" (UniqueName: \"kubernetes.io/projected/c27c447f-f1fa-4d0e-809e-f2945b673c32-kube-api-access-9tjn9\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.293233 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.293216    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sln\" (UniqueName: \"kubernetes.io/projected/b582586d-a43f-4e66-b5c2-2d83f406f0c3-kube-api-access-l4sln\") pod \"service-ca-5fffc97bdf-xhrrk\" (UID: \"b582586d-a43f-4e66-b5c2-2d83f406f0c3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhrrk"
Oct 30 02:42:02.293876 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.293857    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jx4h\" (UniqueName: \"kubernetes.io/projected/cea2f02d-9272-44dc-acd9-d899a2d711a9-kube-api-access-6jx4h\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.294322 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.294304    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlwq\" (UniqueName: \"kubernetes.io/projected/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-kube-api-access-xnlwq\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:42:02.294494 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.294479    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqnr\" (UniqueName: \"kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr\") pod \"periodic-gathering-jbbb7-m2nlh\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") " pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:42:02.373102 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373037    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-netns\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373102 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373067    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-multus-certs\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373102 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373085    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-kubernetes\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373102 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373101    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-sys\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373117    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-lib-modules\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373142    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-conf\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373150    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-netns\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373153    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-multus-certs\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373184    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-sys\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373165    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-cnibin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373222    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-cnibin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373225    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-conf-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373225    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-kubernetes\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373261    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818913a6-9255-4e66-b325-5ced6778ad64-hosts-file\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373273    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-conf\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373285    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-conf-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373325    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-lib-modules\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373365 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373343    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373385    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818913a6-9255-4e66-b325-5ced6778ad64-hosts-file\") pod \"node-resolver-k5blc\" (UID: \"818913a6-9255-4e66-b325-5ced6778ad64\") " pod="openshift-dns/node-resolver-k5blc"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373386    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373413    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-host-slash\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373430    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-system-cni-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373458    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-netns\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373483    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-cnibin\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373499    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9d43c51-c7ff-4751-9c3b-0eeae8d6a355-host-slash\") pod \"iptables-alerter-8l6sk\" (UID: \"c9d43c51-c7ff-4751-9c3b-0eeae8d6a355\") " pod="openshift-network-operator/iptables-alerter-8l6sk"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373506    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373512    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-system-cni-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373529    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-system-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373529    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-run-netns\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373539    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-cnibin\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373543    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-etc-kubernetes\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373553    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d90d1fba-6d54-49b3-8792-36688f2242d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hp7gt\" (UID: \"d90d1fba-6d54-49b3-8792-36688f2242d3\") " pod="openshift-multus/multus-additional-cni-plugins-hp7gt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373583    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-kubelet-config\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373589    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-etc-kubernetes\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373619    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-system-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373640    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-netd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373656    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-kubelet-config\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373688    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-run\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373708    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-var-lib-kubelet\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373722    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-bin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373740    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-netd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373763    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-host\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373781    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-node-log\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373796    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-log-socket\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373819    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-log-socket\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373841    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-var-lib-kubelet\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373845    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-run\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373850    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-node-log\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373849    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-bin\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373863    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180f424e-5860-452d-98b6-c0d4585dd13c-crio-socket\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373876    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1628ebb0-5634-4a42-bca2-b09ecdbfd6be-host\") pod \"node-ca-qb86g\" (UID: \"1628ebb0-5634-4a42-bca2-b09ecdbfd6be\") " pod="openshift-image-registry/node-ca-qb86g"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373899    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373954    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-cni-dir\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373973    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-socket-dir-parent\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.373991    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-kubelet\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374006    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:02.373987 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374013    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/180f424e-5860-452d-98b6-c0d4585dd13c-crio-socket\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374024    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374038    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-hostroot\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374042    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-kubelet\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374054    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374081    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374081    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-multus-socket-dir-parent\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374105    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-hostroot\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374111    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-dbus\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374126    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.374130    4334 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374130    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-slash\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374158    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374170    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374189    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-slash\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.374207    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.874192317 +0000 UTC m=+1.880782553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : secret "metrics-daemon-secret" not found
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374222    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-registration-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374228    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/033b7d72-7e35-47ee-9cf9-d9993d21f4be-dbus\") pod \"global-pull-secret-syncer-6xh6w\" (UID: \"033b7d72-7e35-47ee-9cf9-d9993d21f4be\") " pod="kube-system/global-pull-secret-syncer-6xh6w"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.374252    4334 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374279    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.374339    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.874324445 +0000 UTC m=+1.880914692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374355    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-registration-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374363    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-device-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374392    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-sys-fs\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374403    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-device-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374409    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysctl-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374420    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-systemd-units\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374441    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-sys-fs\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374463    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-bin\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374487    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-cni-bin\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374490    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-systemd\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374513    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-socket-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374537    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374552    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-systemd\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374466    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-systemd-units\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374564    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-socket-dir\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374563    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-k8s-cni-cncf-io\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374592    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-multus\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374604    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-run-k8s-cni-cncf-io\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374611    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-systemd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374614    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05e47302-4160-472b-988d-60585665c3bf-host-var-lib-cni-multus\") pod \"multus-pvqdb\" (UID: \"05e47302-4160-472b-988d-60585665c3bf\") " pod="openshift-multus/multus-pvqdb"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374592    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374633    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-etc-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374635    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-systemd\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374633    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/180f424e-5860-452d-98b6-c0d4585dd13c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374656    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-etc-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374688    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-ovn\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374656    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-run-ovn\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374726    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-kubelet\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374741    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374757    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-var-lib-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374772    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-modprobe-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374787    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-host-kubelet\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374792    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-host\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374799    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c27c447f-f1fa-4d0e-809e-f2945b673c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzccg\" (UID: \"c27c447f-f1fa-4d0e-809e-f2945b673c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzccg"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374814    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cea2f02d-9272-44dc-acd9-d899a2d711a9-var-lib-openvswitch\") pod \"ovnkube-node-zdxqj\" (UID: \"cea2f02d-9272-44dc-acd9-d899a2d711a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374826    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374835    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-host\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374847    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysconfig\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.374870    4334 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374882    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-modprobe-d\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.374905    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:02.874895872 +0000 UTC m=+1.881486121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:02.375651 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.374908    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb013b42-f40c-4df9-bd10-ccb8975e54dc-etc-sysconfig\") pod \"tuned-vj6lt\" (UID: \"fb013b42-f40c-4df9-bd10-ccb8975e54dc\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6lt"
Oct 30 02:42:02.873212 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.873177    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5tmj8_c080eab2-cfb6-4495-ba04-475ad7344a43/serve-healthcheck-canary/0.log"
Oct 30 02:42:02.879259 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.879238    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:02.879338 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.879281    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:02.879338 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:02.879303    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:02.879409 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.879393    4334 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:42:02.879439 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.879405    4334 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:02.879485 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.879467    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:42:03.879435369 +0000 UTC m=+2.886025614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:02.879485 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.879481    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:03.879474282 +0000 UTC m=+2.886064517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : secret "metrics-daemon-secret" not found
Oct 30 02:42:02.879558 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.879487    4334 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:02.879558 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:02.879518    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:03.879506076 +0000 UTC m=+2.886096307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:03.464715 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:03.464681    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-z89rx_289d5912-6d50-4d1a-b359-1db38fd88f08/migrator/0.log"
Oct 30 02:42:03.628016 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:03.627992    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:03.889569 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:03.889494    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:03.889569 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:03.889555    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:03.889733 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:03.889608    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:03.889733 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:03.889637    4334 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:03.889733 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:03.889714    4334 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:03.889831 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:03.889737    4334 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:42:03.889831 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:03.889746    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:42:05.889729936 +0000 UTC m=+4.896320168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:03.889831 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:03.889774    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:05.889757563 +0000 UTC m=+4.896347799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:03.889831 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:03.889801    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs podName:cf492558-e351-45c6-8772-2324e53b4a97 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:05.889791509 +0000 UTC m=+4.896381741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs") pod "network-metrics-daemon-gztvh" (UID: "cf492558-e351-45c6-8772-2324e53b4a97") : secret "metrics-daemon-secret" not found
Oct 30 02:42:05.903664 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:05.903471    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:05.904113 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:05.903678    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:05.904113 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:05.903712    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:05.904113 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:05.903841    4334 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:05.904113 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:05.903880    4334 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:42:05.904113 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:05.903939    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls podName:180f424e-5860-452d-98b6-c0d4585dd13c nodeName:}" failed. No retries permitted until 2026-10-30 02:42:09.903921106 +0000 UTC m=+8.910511339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bvj8c" (UID: "180f424e-5860-452d-98b6-c0d4585dd13c") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:42:05.904113 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:05.903954    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls podName:1436ee1e-5bbf-488b-812b-92c7bc0906a2 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:09.903948141 +0000 UTC m=+8.910538372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls") pod "dns-default-8k7nc" (UID: "1436ee1e-5bbf-488b-812b-92c7bc0906a2") : secret "dns-default-metrics-tls" not found
Oct 30 02:42:05.904411 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:05.904327    4334 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:42:05.907254 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:05.907232    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf492558-e351-45c6-8772-2324e53b4a97-metrics-certs\") pod \"network-metrics-daemon-gztvh\" (UID: \"cf492558-e351-45c6-8772-2324e53b4a97\") " pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:06.062935 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:06.062906    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gztvh"
Oct 30 02:42:06.353509 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:06.353481    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gztvh"]
Oct 30 02:42:06.359644 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:06.359618    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf492558_e351_45c6_8772_2324e53b4a97.slice/crio-2d5eda15d679ba41557c9275c88fa08a9cb9abfbb1856128915b36a3304cb473 WatchSource:0}: Error finding container 2d5eda15d679ba41557c9275c88fa08a9cb9abfbb1856128915b36a3304cb473: Status 404 returned error can't find the container with id 2d5eda15d679ba41557c9275c88fa08a9cb9abfbb1856128915b36a3304cb473
Oct 30 02:42:06.361725 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:06.361708    4334 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:42:07.207189 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:07.207152    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gztvh" event={"ID":"cf492558-e351-45c6-8772-2324e53b4a97","Type":"ContainerStarted","Data":"2d5eda15d679ba41557c9275c88fa08a9cb9abfbb1856128915b36a3304cb473"}
Oct 30 02:42:07.716226 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:07.716201    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:42:08.212680 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:08.212652    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gztvh" event={"ID":"cf492558-e351-45c6-8772-2324e53b4a97","Type":"ContainerStarted","Data":"15c91be683241550ef143768fab280c63f1a6aaa8d8274851eec660f7992b9de"}
Oct 30 02:42:08.213052 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:08.212701    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gztvh" event={"ID":"cf492558-e351-45c6-8772-2324e53b4a97","Type":"ContainerStarted","Data":"b92a909861c6c3abb495a4f01173a6957a860a701b5bf154ac8e03e996b3e1ba"}
Oct 30 02:42:09.132335 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:09.132308    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mkxt8"
Oct 30 02:42:09.933111 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:09.933072    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:09.933529 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:09.933123    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:09.935494 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:09.935469    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/180f424e-5860-452d-98b6-c0d4585dd13c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bvj8c\" (UID: \"180f424e-5860-452d-98b6-c0d4585dd13c\") " pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:09.935591 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:09.935522    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1436ee1e-5bbf-488b-812b-92c7bc0906a2-metrics-tls\") pod \"dns-default-8k7nc\" (UID: \"1436ee1e-5bbf-488b-812b-92c7bc0906a2\") " pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:09.964503 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:09.964466    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:09.968531 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:09.968516    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bvj8c"
Oct 30 02:42:10.277270 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:10.277200    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8k7nc"]
Oct 30 02:42:10.281861 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:10.281838    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1436ee1e_5bbf_488b_812b_92c7bc0906a2.slice/crio-fcdbc0b27dbd0120963ad171e0615f7bf781c2cbb552e9d6a4bb0772d17dafa8 WatchSource:0}: Error finding container fcdbc0b27dbd0120963ad171e0615f7bf781c2cbb552e9d6a4bb0772d17dafa8: Status 404 returned error can't find the container with id fcdbc0b27dbd0120963ad171e0615f7bf781c2cbb552e9d6a4bb0772d17dafa8
Oct 30 02:42:10.290903 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:10.290882    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bvj8c"]
Oct 30 02:42:10.295115 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:10.295093    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180f424e_5860_452d_98b6_c0d4585dd13c.slice/crio-ce5e4ad6c1f03492d890bd294e529df68a2a5fa96bfec3709cc589f560141842 WatchSource:0}: Error finding container ce5e4ad6c1f03492d890bd294e529df68a2a5fa96bfec3709cc589f560141842: Status 404 returned error can't find the container with id ce5e4ad6c1f03492d890bd294e529df68a2a5fa96bfec3709cc589f560141842
Oct 30 02:42:11.225952 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:11.225919    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8k7nc" event={"ID":"1436ee1e-5bbf-488b-812b-92c7bc0906a2","Type":"ContainerStarted","Data":"fcdbc0b27dbd0120963ad171e0615f7bf781c2cbb552e9d6a4bb0772d17dafa8"}
Oct 30 02:42:11.227636 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:11.227598    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvj8c" event={"ID":"180f424e-5860-452d-98b6-c0d4585dd13c","Type":"ContainerStarted","Data":"7ebd20a2eae5d83570d681db9ef6a04d5de51412fecc292dc0cf80c5bab1b185"}
Oct 30 02:42:11.227756 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:11.227663    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvj8c" event={"ID":"180f424e-5860-452d-98b6-c0d4585dd13c","Type":"ContainerStarted","Data":"ce5e4ad6c1f03492d890bd294e529df68a2a5fa96bfec3709cc589f560141842"}
Oct 30 02:42:11.319188 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:11.319160    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:11.319429 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:11.319409    4334 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:42:12.238889 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:12.238847    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8k7nc" event={"ID":"1436ee1e-5bbf-488b-812b-92c7bc0906a2","Type":"ContainerStarted","Data":"c49e913bf0b475d04e38ca728dd00b6e168c5b834e5bbf02247d14da2c45bdea"}
Oct 30 02:42:12.239300 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:12.238954    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8k7nc" event={"ID":"1436ee1e-5bbf-488b-812b-92c7bc0906a2","Type":"ContainerStarted","Data":"a63d95a6164bd444a6142d668efa65a5575fce22c0b79b04ee369bcdbc1b80bf"}
Oct 30 02:42:12.239300 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:12.238988    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:12.241229 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:12.241204    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvj8c" event={"ID":"180f424e-5860-452d-98b6-c0d4585dd13c","Type":"ContainerStarted","Data":"a730fd9a31128874d3733abbdb6f7ec2a7f315a9d7bc10866f93080acf528c10"}
Oct 30 02:42:13.246376 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:13.246342    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bvj8c" event={"ID":"180f424e-5860-452d-98b6-c0d4585dd13c","Type":"ContainerStarted","Data":"adf34af36de380a200aa461f2a52af858008eeca1d8fc5149f5642f726bbcfb1"}
Oct 30 02:42:14.511927 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.511892    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-j8m67"]
Oct 30 02:42:14.514351 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.514330    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:14.523707 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.523685    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"]
Oct 30 02:42:14.525571 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.525546    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"
Oct 30 02:42:14.536784 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.536765    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 02:42:14.536784 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.536764    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 02:42:14.537180 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.537166    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-8qqwz\""
Oct 30 02:42:14.549188 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.549168    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"]
Oct 30 02:42:14.549567 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.549545    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j8m67"]
Oct 30 02:42:14.604367 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.604343    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d86776dcf-bqt2x"]
Oct 30 02:42:14.606494 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.606477    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.638439 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.638415    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:42:14.638573 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.638424    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:42:14.642038 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.639838    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d86776dcf-bqt2x"]
Oct 30 02:42:14.649744 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.649724    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:42:14.649842 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.649818    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 02:42:14.649890 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.649866    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:42:14.649890 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.649884    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8xpcs\""
Oct 30 02:42:14.649969 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.649956    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k5nlz\""
Oct 30 02:42:14.664848 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.664825    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a14bb01-c443-4ddb-910f-2ef20f279ba5-tmp\") pod \"downloads-6dd789c8b7-j8m67\" (UID: \"2a14bb01-c443-4ddb-910f-2ef20f279ba5\") " pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:14.664937 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.664851    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtzb\" (UniqueName: \"kubernetes.io/projected/2a14bb01-c443-4ddb-910f-2ef20f279ba5-kube-api-access-fgtzb\") pod \"downloads-6dd789c8b7-j8m67\" (UID: \"2a14bb01-c443-4ddb-910f-2ef20f279ba5\") " pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:14.664937 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.664884    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/becf929b-932d-4100-bc01-0adec847862b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hds4n\" (UID: \"becf929b-932d-4100-bc01-0adec847862b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"
Oct 30 02:42:14.766098 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766038    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9869074-3251-4428-852a-30d9058ec7ab-installation-pull-secrets\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.766098 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766067    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9869074-3251-4428-852a-30d9058ec7ab-image-registry-private-configuration\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.766098 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766089    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtzb\" (UniqueName: \"kubernetes.io/projected/2a14bb01-c443-4ddb-910f-2ef20f279ba5-kube-api-access-fgtzb\") pod \"downloads-6dd789c8b7-j8m67\" (UID: \"2a14bb01-c443-4ddb-910f-2ef20f279ba5\") " pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:14.766273 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766140    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9869074-3251-4428-852a-30d9058ec7ab-registry-certificates\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.766337 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766319    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9869074-3251-4428-852a-30d9058ec7ab-trusted-ca\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.766377 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766348    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66wv\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-kube-api-access-t66wv\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.766377 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766366    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-bound-sa-token\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.766466 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766393    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a14bb01-c443-4ddb-910f-2ef20f279ba5-tmp\") pod \"downloads-6dd789c8b7-j8m67\" (UID: \"2a14bb01-c443-4ddb-910f-2ef20f279ba5\") " pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:14.766580 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766563    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-registry-tls\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.766617 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766598    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/becf929b-932d-4100-bc01-0adec847862b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hds4n\" (UID: \"becf929b-932d-4100-bc01-0adec847862b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"
Oct 30 02:42:14.766650 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766619    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9869074-3251-4428-852a-30d9058ec7ab-ca-trust-extracted\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.766742 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.766726    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a14bb01-c443-4ddb-910f-2ef20f279ba5-tmp\") pod \"downloads-6dd789c8b7-j8m67\" (UID: \"2a14bb01-c443-4ddb-910f-2ef20f279ba5\") " pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:14.769054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.769034    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/becf929b-932d-4100-bc01-0adec847862b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hds4n\" (UID: \"becf929b-932d-4100-bc01-0adec847862b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"
Oct 30 02:42:14.785977 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.785948    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtzb\" (UniqueName: \"kubernetes.io/projected/2a14bb01-c443-4ddb-910f-2ef20f279ba5-kube-api-access-fgtzb\") pod \"downloads-6dd789c8b7-j8m67\" (UID: \"2a14bb01-c443-4ddb-910f-2ef20f279ba5\") " pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:14.824525 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.824502    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:14.834515 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.834491    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"
Oct 30 02:42:14.868879 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.868837    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9869074-3251-4428-852a-30d9058ec7ab-installation-pull-secrets\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.869069 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.868891    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9869074-3251-4428-852a-30d9058ec7ab-image-registry-private-configuration\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.869069 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.868935    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9869074-3251-4428-852a-30d9058ec7ab-registry-certificates\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.869069 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.868978    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9869074-3251-4428-852a-30d9058ec7ab-trusted-ca\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.869069 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.869004    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66wv\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-kube-api-access-t66wv\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.869069 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.869029    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-bound-sa-token\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.869069 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.869070    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-registry-tls\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.869413 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.869111    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9869074-3251-4428-852a-30d9058ec7ab-ca-trust-extracted\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.869585 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.869562    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9869074-3251-4428-852a-30d9058ec7ab-ca-trust-extracted\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.870873 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.870842    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9869074-3251-4428-852a-30d9058ec7ab-registry-certificates\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.871015 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.870842    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9869074-3251-4428-852a-30d9058ec7ab-trusted-ca\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.872726 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.872526    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-registry-tls\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.873050 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.872940    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9869074-3251-4428-852a-30d9058ec7ab-installation-pull-secrets\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.878411 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.877555    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9869074-3251-4428-852a-30d9058ec7ab-image-registry-private-configuration\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.889083 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.888339    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-bound-sa-token\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.889547 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.889497    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66wv\" (UniqueName: \"kubernetes.io/projected/f9869074-3251-4428-852a-30d9058ec7ab-kube-api-access-t66wv\") pod \"image-registry-5d86776dcf-bqt2x\" (UID: \"f9869074-3251-4428-852a-30d9058ec7ab\") " pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:14.916915 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:14.916851    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:15.144322 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:15.144250    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j8m67"]
Oct 30 02:42:15.147364 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:15.147337    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"]
Oct 30 02:42:15.153074 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:15.153042    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a14bb01_c443_4ddb_910f_2ef20f279ba5.slice/crio-9045dee8962778e62a4af0ac88b89eaf95a7809e553c6fbc1b90bb5903baadf2 WatchSource:0}: Error finding container 9045dee8962778e62a4af0ac88b89eaf95a7809e553c6fbc1b90bb5903baadf2: Status 404 returned error can't find the container with id 9045dee8962778e62a4af0ac88b89eaf95a7809e553c6fbc1b90bb5903baadf2
Oct 30 02:42:15.161220 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:15.161198    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecf929b_932d_4100_bc01_0adec847862b.slice/crio-6dac4c6d08462140f586b5c8577941f6cf4af64f6d37a6899e3e090334a23ee6 WatchSource:0}: Error finding container 6dac4c6d08462140f586b5c8577941f6cf4af64f6d37a6899e3e090334a23ee6: Status 404 returned error can't find the container with id 6dac4c6d08462140f586b5c8577941f6cf4af64f6d37a6899e3e090334a23ee6
Oct 30 02:42:15.209605 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:15.209579    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d86776dcf-bqt2x"]
Oct 30 02:42:15.213440 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:15.213412    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9869074_3251_4428_852a_30d9058ec7ab.slice/crio-ce190e36ee7e6edcc203e8e4c92a0d03fa4b0e6f2bb7e607ed05fd4209c17c5c WatchSource:0}: Error finding container ce190e36ee7e6edcc203e8e4c92a0d03fa4b0e6f2bb7e607ed05fd4209c17c5c: Status 404 returned error can't find the container with id ce190e36ee7e6edcc203e8e4c92a0d03fa4b0e6f2bb7e607ed05fd4209c17c5c
Oct 30 02:42:15.254042 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:15.254016    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x" event={"ID":"f9869074-3251-4428-852a-30d9058ec7ab","Type":"ContainerStarted","Data":"ce190e36ee7e6edcc203e8e4c92a0d03fa4b0e6f2bb7e607ed05fd4209c17c5c"}
Oct 30 02:42:15.255162 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:15.255135    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n" event={"ID":"becf929b-932d-4100-bc01-0adec847862b","Type":"ContainerStarted","Data":"6dac4c6d08462140f586b5c8577941f6cf4af64f6d37a6899e3e090334a23ee6"}
Oct 30 02:42:15.256262 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:15.256242    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j8m67" event={"ID":"2a14bb01-c443-4ddb-910f-2ef20f279ba5","Type":"ContainerStarted","Data":"9045dee8962778e62a4af0ac88b89eaf95a7809e553c6fbc1b90bb5903baadf2"}
Oct 30 02:42:16.262768 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:16.262731    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n" event={"ID":"becf929b-932d-4100-bc01-0adec847862b","Type":"ContainerStarted","Data":"1926c251689fdf5941b81f7c92784958f69884d19c7a9fa932eee8ea17c948c1"}
Oct 30 02:42:16.263256 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:16.262916    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"
Oct 30 02:42:16.265095 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:16.265071    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x" event={"ID":"f9869074-3251-4428-852a-30d9058ec7ab","Type":"ContainerStarted","Data":"37d1e7a1ffd533d4ee2548d148dc4444a0dc17fe64256308b61063acf403d047"}
Oct 30 02:42:16.265237 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:16.265220    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:16.268788 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:16.268766    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n"
Oct 30 02:42:16.299321 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:16.299258    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hds4n" podStartSLOduration=1.401649746 podStartE2EDuration="2.299240822s" totalImagesPullingTime="897.591076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:16.298144844 +0000 UTC m=+15.304735099" watchObservedRunningTime="2026-10-30 02:42:16.299240822 +0000 UTC m=+15.305831077"
Oct 30 02:42:16.376115 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:16.376021    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x" podStartSLOduration=2.376006263 podStartE2EDuration="2.376006263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:16.374906701 +0000 UTC m=+15.381496957" watchObservedRunningTime="2026-10-30 02:42:16.376006263 +0000 UTC m=+15.382596519"
Oct 30 02:42:20.985402 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:20.985366    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:42:22.000100 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.000073    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c4wz5"]
Oct 30 02:42:22.003305 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.003279    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.034787 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.034758    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:42:22.034920 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.034862    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:42:22.034996 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.034919    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f69xl\""
Oct 30 02:42:22.107617 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.107587    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:42:22.107617 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.107590    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:42:22.107817 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.107771    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:42:22.107817 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.107790    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:42:22.126119 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126094    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-root\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.126264 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126132    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-sys\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.126264 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126161    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-textfile\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.126264 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126186    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.126383 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126314    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.126383 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126364    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-tls\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.126511 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126476    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5xg\" (UniqueName: \"kubernetes.io/projected/ce6fe84e-ade4-48a8-8737-5767fe363a66-kube-api-access-bx5xg\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.126511 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126506    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-wtmp\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.126604 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.126533    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce6fe84e-ade4-48a8-8737-5767fe363a66-metrics-client-ca\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.227747 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.227709    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.227889 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.227784    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-tls\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.227889 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.227828    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5xg\" (UniqueName: \"kubernetes.io/projected/ce6fe84e-ade4-48a8-8737-5767fe363a66-kube-api-access-bx5xg\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.227889 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.227856    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-wtmp\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228070 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.227908    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce6fe84e-ade4-48a8-8737-5767fe363a66-metrics-client-ca\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228070 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.227970    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-root\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228070 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.227991    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-wtmp\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228070 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.227997    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-sys\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228070 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.228027    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-textfile\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228327 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.228112    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-sys\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228327 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.228144    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228327 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.228255    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce6fe84e-ade4-48a8-8737-5767fe363a66-root\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228510 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.228358    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228510 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.228470    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-textfile\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.228594 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.228510    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce6fe84e-ade4-48a8-8737-5767fe363a66-metrics-client-ca\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.230396 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.230373    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-tls\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.231189 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.231164    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce6fe84e-ade4-48a8-8737-5767fe363a66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.245657 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.245606    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5xg\" (UniqueName: \"kubernetes.io/projected/ce6fe84e-ade4-48a8-8737-5767fe363a66-kube-api-access-bx5xg\") pod \"node-exporter-c4wz5\" (UID: \"ce6fe84e-ade4-48a8-8737-5767fe363a66\") " pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.248917 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.248887    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8k7nc"
Oct 30 02:42:22.323817 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:22.323750    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4wz5"
Oct 30 02:42:22.338666 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:22.338641    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6fe84e_ade4_48a8_8737_5767fe363a66.slice/crio-3e81c2e1776ad413afb1003b304f5f9cad7ba40050780ed3236c465665a0510d WatchSource:0}: Error finding container 3e81c2e1776ad413afb1003b304f5f9cad7ba40050780ed3236c465665a0510d: Status 404 returned error can't find the container with id 3e81c2e1776ad413afb1003b304f5f9cad7ba40050780ed3236c465665a0510d
Oct 30 02:42:23.294036 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:23.293994    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4wz5" event={"ID":"ce6fe84e-ade4-48a8-8737-5767fe363a66","Type":"ContainerStarted","Data":"3e81c2e1776ad413afb1003b304f5f9cad7ba40050780ed3236c465665a0510d"}
Oct 30 02:42:24.299873 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:24.299832    4334 generic.go:356] "Generic (PLEG): container finished" podID="ce6fe84e-ade4-48a8-8737-5767fe363a66" containerID="6ffba05c6180d29afa547815659b3236576eab2bbe474af0125afa52169ebbf4" exitCode=0
Oct 30 02:42:24.300380 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:24.299908    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4wz5" event={"ID":"ce6fe84e-ade4-48a8-8737-5767fe363a66","Type":"ContainerDied","Data":"6ffba05c6180d29afa547815659b3236576eab2bbe474af0125afa52169ebbf4"}
Oct 30 02:42:25.306111 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:25.306077    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4wz5" event={"ID":"ce6fe84e-ade4-48a8-8737-5767fe363a66","Type":"ContainerStarted","Data":"389e8d5c2d3d9d247ae54846200f0109093101c9653cc7235edb05e71d780554"}
Oct 30 02:42:25.306534 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:25.306137    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4wz5" event={"ID":"ce6fe84e-ade4-48a8-8737-5767fe363a66","Type":"ContainerStarted","Data":"af1775664b5c5c78e451d3f6ec90ac7d2ba1e804c658a7e2475763526117dcb7"}
Oct 30 02:42:25.367193 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:25.367136    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c4wz5" podStartSLOduration=3.386024224 podStartE2EDuration="4.36712206s" totalImagesPullingTime="905.086117ms" totalInitContainerRuntime="76.011719ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:25.355763062 +0000 UTC m=+24.362353315" watchObservedRunningTime="2026-10-30 02:42:25.36712206 +0000 UTC m=+24.373712363"
Oct 30 02:42:31.331572 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:31.331532    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j8m67" event={"ID":"2a14bb01-c443-4ddb-910f-2ef20f279ba5","Type":"ContainerStarted","Data":"e2d429b964e96737e72613ef4de0e030a35cd7dabead5d58e4e6804e14a9a1a8"}
Oct 30 02:42:31.332130 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:31.332109    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:31.349394 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:31.349370    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-j8m67"
Oct 30 02:42:31.391535 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:31.391469    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-j8m67" podStartSLOduration=1.737456493 podStartE2EDuration="17.391435697s" totalImagesPullingTime="15.653979204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:31.379868424 +0000 UTC m=+30.386458681" watchObservedRunningTime="2026-10-30 02:42:31.391435697 +0000 UTC m=+30.398025930"
Oct 30 02:42:32.117769 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:32.117740    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zdxqj"
Oct 30 02:42:35.619179 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.619145    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-664cf94d99-psvrh"]
Oct 30 02:42:35.651266 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.651229    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664cf94d99-psvrh"]
Oct 30 02:42:35.651458 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.651331    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.665554 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.665528    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 02:42:35.665721 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.665560    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 02:42:35.665868 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.665568    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 02:42:35.676667 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.676638    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 02:42:35.676667 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.676656    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 02:42:35.688353 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.688330    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z7sc4\""
Oct 30 02:42:35.700044 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.700022    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 02:42:35.730754 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.730722    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-tmp\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.730872 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.730763    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-oauth-serving-cert\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.730872 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.730792    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-service-ca\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.730972 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.730919    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-serving-cert\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.730972 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.730953    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-trusted-ca-bundle\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.731078 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.731034    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-oauth-config\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.731078 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.731074    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmc4\" (UniqueName: \"kubernetes.io/projected/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-kube-api-access-lgmc4\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.731175 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.731112    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-config\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.831501 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831470    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-tmp\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.831501 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831505    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-oauth-serving-cert\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.831732 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831526    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-service-ca\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.831732 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831555    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-serving-cert\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.831732 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831582    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-trusted-ca-bundle\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.831732 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831640    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-oauth-config\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.831732 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831664    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmc4\" (UniqueName: \"kubernetes.io/projected/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-kube-api-access-lgmc4\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.831732 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831694    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-config\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.832110 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.831953    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-tmp\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.832342 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.832317    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-oauth-serving-cert\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.833013 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.832986    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-service-ca\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.833207 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.833045    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-config\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.833207 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.833073    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-trusted-ca-bundle\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.834779 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.834754    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-oauth-config\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.834891 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.834754    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-serving-cert\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.850244 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.850220    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmc4\" (UniqueName: \"kubernetes.io/projected/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-kube-api-access-lgmc4\") pod \"console-664cf94d99-psvrh\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") " pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:35.984495 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:35.984442    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:36.313781 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:36.313691    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664cf94d99-psvrh"]
Oct 30 02:42:36.319673 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:42:36.319638    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce6040c_4eb2_4df1_81cd_0b18ad2c050b.slice/crio-1ef3a8231dc40a1ba800a733065c565fbcbb6d36712e36f06e89092f933dfe30 WatchSource:0}: Error finding container 1ef3a8231dc40a1ba800a733065c565fbcbb6d36712e36f06e89092f933dfe30: Status 404 returned error can't find the container with id 1ef3a8231dc40a1ba800a733065c565fbcbb6d36712e36f06e89092f933dfe30
Oct 30 02:42:36.353302 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:36.353263    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664cf94d99-psvrh" event={"ID":"cce6040c-4eb2-4df1-81cd-0b18ad2c050b","Type":"ContainerStarted","Data":"1ef3a8231dc40a1ba800a733065c565fbcbb6d36712e36f06e89092f933dfe30"}
Oct 30 02:42:37.276204 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:37.276175    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d86776dcf-bqt2x"
Oct 30 02:42:37.720235 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:37.720202    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v7c87"
Oct 30 02:42:40.370859 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:40.370827    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664cf94d99-psvrh" event={"ID":"cce6040c-4eb2-4df1-81cd-0b18ad2c050b","Type":"ContainerStarted","Data":"83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173"}
Oct 30 02:42:40.419297 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:40.419253    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-664cf94d99-psvrh" podStartSLOduration=2.00956212 podStartE2EDuration="5.419241302s" totalImagesPullingTime="3.409679182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:40.407987747 +0000 UTC m=+39.414578002" watchObservedRunningTime="2026-10-30 02:42:40.419241302 +0000 UTC m=+39.425831555"
Oct 30 02:42:45.984792 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:45.984670    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:45.984792 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:45.984733    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:45.989403 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:45.989379    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:46.393399 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:42:46.393331    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:42:50.980841 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:42:50.980803    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:43:02.631802 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.631765    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"]
Oct 30 02:43:02.689852 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.689823    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"]
Oct 30 02:43:02.690025 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.689948    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.706173 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.706149    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 02:43:02.706347 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.706245    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 02:43:02.706347 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.706243    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 02:43:02.706347 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.706268    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 02:43:02.706347 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.706293    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tdpcb\""
Oct 30 02:43:02.715716 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.715695    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgkg\" (UniqueName: \"kubernetes.io/projected/473e27f6-e488-472b-a0b1-4ede62f08ba1-kube-api-access-shgkg\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.715805 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.715721    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-federate-client-tls\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.715805 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.715754    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-serving-certs-ca-bundle\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.715883 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.715810    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-metrics-client-ca\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.715883 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.715839    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.715975 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.715960    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-secret-telemeter-client\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.716032 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.716018    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.716069 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.716042    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-telemeter-client-tls\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.748645 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.748620    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 02:43:02.760815 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.760796    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 02:43:02.816959 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.816936    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-serving-certs-ca-bundle\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.817056 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.816969    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-metrics-client-ca\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.817056 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.816987    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.817056 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.817006    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-secret-telemeter-client\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.817056 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.817027    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.817259 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.817133    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-telemeter-client-tls\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.817259 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.817186    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgkg\" (UniqueName: \"kubernetes.io/projected/473e27f6-e488-472b-a0b1-4ede62f08ba1-kube-api-access-shgkg\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.817259 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.817209    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-federate-client-tls\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.817810 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.817790    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-metrics-client-ca\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.818376 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.818348    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-serving-certs-ca-bundle\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.818489 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.818373    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/473e27f6-e488-472b-a0b1-4ede62f08ba1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.819695 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.819670    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-telemeter-client-tls\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.819695 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.819683    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-secret-telemeter-client\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.819824 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.819712    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.819824 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.819724    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/473e27f6-e488-472b-a0b1-4ede62f08ba1-federate-client-tls\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:02.834421 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:02.834402    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgkg\" (UniqueName: \"kubernetes.io/projected/473e27f6-e488-472b-a0b1-4ede62f08ba1-kube-api-access-shgkg\") pod \"telemeter-client-bd95bbcdd-n9zhl\" (UID: \"473e27f6-e488-472b-a0b1-4ede62f08ba1\") " pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:03.042481 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:03.042442    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"
Oct 30 02:43:03.406290 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:03.406259    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl"]
Oct 30 02:43:03.423749 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:43:03.423724    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473e27f6_e488_472b_a0b1_4ede62f08ba1.slice/crio-e792ce4c4df43ec591ed36ddc5b001e4b3f2468b54ce05250fb8c5e7c8e22e8f WatchSource:0}: Error finding container e792ce4c4df43ec591ed36ddc5b001e4b3f2468b54ce05250fb8c5e7c8e22e8f: Status 404 returned error can't find the container with id e792ce4c4df43ec591ed36ddc5b001e4b3f2468b54ce05250fb8c5e7c8e22e8f
Oct 30 02:43:03.445370 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:03.445348    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl" event={"ID":"473e27f6-e488-472b-a0b1-4ede62f08ba1","Type":"ContainerStarted","Data":"e792ce4c4df43ec591ed36ddc5b001e4b3f2468b54ce05250fb8c5e7c8e22e8f"}
Oct 30 02:43:05.456532 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:05.456499    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl" event={"ID":"473e27f6-e488-472b-a0b1-4ede62f08ba1","Type":"ContainerStarted","Data":"656ad1266d3c0cad0244e46bc38ebbdbd2c8e22eadaff0c6e42fd0cd8421ff19"}
Oct 30 02:43:07.465133 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.465101    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl" event={"ID":"473e27f6-e488-472b-a0b1-4ede62f08ba1","Type":"ContainerStarted","Data":"48e3fa7be66aa7ef82b116e2d1fba31758eb01366c82c6c58a43375b54de2029"}
Oct 30 02:43:07.465660 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.465152    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl" event={"ID":"473e27f6-e488-472b-a0b1-4ede62f08ba1","Type":"ContainerStarted","Data":"b0b9f6a75815038baf0be9420c52850fa156999c26211f39a9a16455b51ace77"}
Oct 30 02:43:07.528939 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.528891    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-bd95bbcdd-n9zhl" podStartSLOduration=2.241237802 podStartE2EDuration="5.528878287s" totalImagesPullingTime="3.287640485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:43:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:43:07.52848145 +0000 UTC m=+66.535071703" watchObservedRunningTime="2026-10-30 02:43:07.528878287 +0000 UTC m=+66.535468540"
Oct 30 02:43:07.682042 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.682011    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-bcb4cf79f-fqtvl"]
Oct 30 02:43:07.705116 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.705087    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bcb4cf79f-fqtvl"]
Oct 30 02:43:07.705250 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.705198    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.756172 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.756113    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6575d45-a32f-491e-8c20-56fbf6bd8604-tmp\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.756172 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.756166    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-config\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.756316 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.756248    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-oauth-config\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.756316 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.756276    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-trusted-ca-bundle\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.756316 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.756292    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-oauth-serving-cert\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.756413 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.756402    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb9dj\" (UniqueName: \"kubernetes.io/projected/b6575d45-a32f-491e-8c20-56fbf6bd8604-kube-api-access-vb9dj\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.756466 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.756419    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-serving-cert\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.756524 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.756510    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-service-ca\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857378 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857348    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6575d45-a32f-491e-8c20-56fbf6bd8604-tmp\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857535 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857387    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-config\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857535 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857417    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-oauth-config\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857535 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857442    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-trusted-ca-bundle\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857535 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857486    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-oauth-serving-cert\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857705 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857551    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb9dj\" (UniqueName: \"kubernetes.io/projected/b6575d45-a32f-491e-8c20-56fbf6bd8604-kube-api-access-vb9dj\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857705 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857579    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-serving-cert\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857705 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857606    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-service-ca\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.857873 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.857854    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6575d45-a32f-491e-8c20-56fbf6bd8604-tmp\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.858266 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.858240    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-config\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.858266 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.858252    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-oauth-serving-cert\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.858383 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.858286    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-service-ca\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.858510 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.858490    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-trusted-ca-bundle\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.860154 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.860130    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-oauth-config\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.860222 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.860171    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-serving-cert\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:07.875501 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:07.875483    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb9dj\" (UniqueName: \"kubernetes.io/projected/b6575d45-a32f-491e-8c20-56fbf6bd8604-kube-api-access-vb9dj\") pod \"console-bcb4cf79f-fqtvl\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") " pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:08.029838 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:08.029760    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:08.346933 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:08.346863    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bcb4cf79f-fqtvl"]
Oct 30 02:43:08.351575 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:43:08.351537    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6575d45_a32f_491e_8c20_56fbf6bd8604.slice/crio-75d73395e374ce7a1484d067572cc41a1b1b3d304a3d6026f1ea9dba3b2f3941 WatchSource:0}: Error finding container 75d73395e374ce7a1484d067572cc41a1b1b3d304a3d6026f1ea9dba3b2f3941: Status 404 returned error can't find the container with id 75d73395e374ce7a1484d067572cc41a1b1b3d304a3d6026f1ea9dba3b2f3941
Oct 30 02:43:08.469798 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:08.469761    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcb4cf79f-fqtvl" event={"ID":"b6575d45-a32f-491e-8c20-56fbf6bd8604","Type":"ContainerStarted","Data":"2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10"}
Oct 30 02:43:08.470219 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:08.469821    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcb4cf79f-fqtvl" event={"ID":"b6575d45-a32f-491e-8c20-56fbf6bd8604","Type":"ContainerStarted","Data":"75d73395e374ce7a1484d067572cc41a1b1b3d304a3d6026f1ea9dba3b2f3941"}
Oct 30 02:43:08.508384 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:08.508331    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-bcb4cf79f-fqtvl" podStartSLOduration=1.508317235 podStartE2EDuration="1.508317235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:43:08.506819934 +0000 UTC m=+67.513410189" watchObservedRunningTime="2026-10-30 02:43:08.508317235 +0000 UTC m=+67.514907490"
Oct 30 02:43:18.030511 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:18.030477    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:18.030996 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:18.030551    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:18.035268 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:18.035244    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:18.506922 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:18.506893    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:43:18.569532 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:18.569493    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664cf94d99-psvrh"]
Oct 30 02:43:20.977839 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:43:20.977800    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:43:29.669859 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.669825    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jbbb7-m2nlh"]
Oct 30 02:43:29.670234 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.670059    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh" podUID="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" containerName="insights-gathering" containerID="cri-o://e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74" gracePeriod=30
Oct 30 02:43:29.833301 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.833271    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-zvcd8-dqps8"]
Oct 30 02:43:29.837174 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.837152    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:29.853261 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.853198    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zvcd8-dqps8"]
Oct 30 02:43:29.908096 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.908078    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:43:29.927991 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.927948    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f962975-75bf-409b-b0e5-a340780bffa5-archives-path\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:29.927991 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.927981    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwjx\" (UniqueName: \"kubernetes.io/projected/4f962975-75bf-409b-b0e5-a340780bffa5-kube-api-access-4fwjx\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:29.928106 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.928018    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-service-ca-bundle\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:29.928149 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:29.928129    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-trusted-ca-bundle\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.028893 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.028869    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path\") pod \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") "
Oct 30 02:43:30.029027 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.028912    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle\") pod \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") "
Oct 30 02:43:30.029027 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.028949    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle\") pod \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") "
Oct 30 02:43:30.029027 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029013    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr\" (UniqueName: \"kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr\") pod \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\" (UID: \"5264ed23-be36-4ef6-bdf7-3d694ab4ec19\") "
Oct 30 02:43:30.029190 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029126    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-trusted-ca-bundle\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.029251 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029187    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f962975-75bf-409b-b0e5-a340780bffa5-archives-path\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.029251 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029220    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwjx\" (UniqueName: \"kubernetes.io/projected/4f962975-75bf-409b-b0e5-a340780bffa5-kube-api-access-4fwjx\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.029353 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029291    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-service-ca-bundle\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.029353 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029218    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path" pod "5264ed23-be36-4ef6-bdf7-3d694ab4ec19" (UID: "5264ed23-be36-4ef6-bdf7-3d694ab4ec19"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:43:30.029443 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029372    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle" pod "5264ed23-be36-4ef6-bdf7-3d694ab4ec19" (UID: "5264ed23-be36-4ef6-bdf7-3d694ab4ec19"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:30.029443 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029311    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle" pod "5264ed23-be36-4ef6-bdf7-3d694ab4ec19" (UID: "5264ed23-be36-4ef6-bdf7-3d694ab4ec19"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:30.029634 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029614    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f962975-75bf-409b-b0e5-a340780bffa5-archives-path\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.029925 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.029905    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-service-ca-bundle\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.030182 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.030165    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-trusted-ca-bundle\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.031298 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.031273    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr" pod "5264ed23-be36-4ef6-bdf7-3d694ab4ec19" (UID: "5264ed23-be36-4ef6-bdf7-3d694ab4ec19"). InnerVolumeSpecName "kube-api-access-rtqnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:43:30.047409 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.047381    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwjx\" (UniqueName: \"kubernetes.io/projected/4f962975-75bf-409b-b0e5-a340780bffa5-kube-api-access-4fwjx\") pod \"periodic-gathering-zvcd8-dqps8\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") " pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.129896 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.129865    4334 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-archives-path\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:30.129896 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.129889    4334 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-service-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:30.129896 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.129901    4334 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-trusted-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:30.130083 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.129910    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqnr\" (UniqueName: \"kubernetes.io/projected/5264ed23-be36-4ef6-bdf7-3d694ab4ec19-kube-api-access-rtqnr\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:30.149817 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.149783    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:43:30.438503 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.438473    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zvcd8-dqps8"]
Oct 30 02:43:30.443242 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:43:30.443217    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f962975_75bf_409b_b0e5_a340780bffa5.slice/crio-0c7ee2dbbcdb6b2efc39319ee02afe92b99d999d0a87b1f4ea1a7ab0bdee4473 WatchSource:0}: Error finding container 0c7ee2dbbcdb6b2efc39319ee02afe92b99d999d0a87b1f4ea1a7ab0bdee4473: Status 404 returned error can't find the container with id 0c7ee2dbbcdb6b2efc39319ee02afe92b99d999d0a87b1f4ea1a7ab0bdee4473
Oct 30 02:43:30.543686 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.543652    4334 generic.go:356] "Generic (PLEG): container finished" podID="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" containerID="e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74" exitCode=2
Oct 30 02:43:30.543855 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.543704    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh"
Oct 30 02:43:30.543855 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.543746    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh" event={"ID":"5264ed23-be36-4ef6-bdf7-3d694ab4ec19","Type":"ContainerDied","Data":"e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74"}
Oct 30 02:43:30.543855 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.543813    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jbbb7-m2nlh" event={"ID":"5264ed23-be36-4ef6-bdf7-3d694ab4ec19","Type":"ContainerDied","Data":"c2bd19973fbd0146c464885e597731726138d9a4cc61a2a0d651c1559a9e8e3c"}
Oct 30 02:43:30.543855 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.543841    4334 scope.go:122] "RemoveContainer" containerID="e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74"
Oct 30 02:43:30.545540 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.545511    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zvcd8-dqps8" event={"ID":"4f962975-75bf-409b-b0e5-a340780bffa5","Type":"ContainerStarted","Data":"b0226a0455f5abfae7c5b2bef1a4f9632016c8b409c749fd92301b8ba73681f4"}
Oct 30 02:43:30.545642 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.545554    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zvcd8-dqps8" event={"ID":"4f962975-75bf-409b-b0e5-a340780bffa5","Type":"ContainerStarted","Data":"0c7ee2dbbcdb6b2efc39319ee02afe92b99d999d0a87b1f4ea1a7ab0bdee4473"}
Oct 30 02:43:30.553072 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.552956    4334 scope.go:122] "RemoveContainer" containerID="e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74"
Oct 30 02:43:30.553360 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:43:30.553332    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74\": container with ID starting with e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74 not found: ID does not exist" containerID="e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74"
Oct 30 02:43:30.553441 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.553367    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74"} err="failed to get container status \"e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74\": rpc error: code = NotFound desc = could not find container \"e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74\": container with ID starting with e0ac7c854702c28446bdeaac2a5d12108d30fb206915cf7c52ed7bf52a081b74 not found: ID does not exist"
Oct 30 02:43:30.593458 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.593425    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jbbb7-m2nlh"]
Oct 30 02:43:30.595719 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.595700    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-jbbb7-m2nlh"]
Oct 30 02:43:30.631507 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:30.631469    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-zvcd8-dqps8" podStartSLOduration=1.63144255 podStartE2EDuration="1.63144255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:43:30.630185427 +0000 UTC m=+89.636775683" watchObservedRunningTime="2026-10-30 02:43:30.63144255 +0000 UTC m=+89.638032806"
Oct 30 02:43:31.157763 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:31.157734    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" path="/var/lib/kubelet/pods/5264ed23-be36-4ef6-bdf7-3d694ab4ec19/volumes"
Oct 30 02:43:33.267947 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.267915    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-545cf8b9c5-b5tb6"]
Oct 30 02:43:33.268304 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.268155    4334 cpu_manager.go:414] "containerMap: removing container" podUID="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" containerName="insights-gathering"
Oct 30 02:43:33.268304 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.268165    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" containerName="insights-gathering"
Oct 30 02:43:33.268304 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.268221    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="5264ed23-be36-4ef6-bdf7-3d694ab4ec19" containerName="insights-gathering"
Oct 30 02:43:33.271054 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.271034    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.288620 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.288592    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-545cf8b9c5-b5tb6"]
Oct 30 02:43:33.455945 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.455913    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44lp\" (UniqueName: \"kubernetes.io/projected/434cbe69-1f14-42f2-bd9c-446dd3a786fb-kube-api-access-d44lp\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.455945 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.455947    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-oauth-serving-cert\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.456110 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.455973    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-oauth-config\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.456110 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.456041    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/434cbe69-1f14-42f2-bd9c-446dd3a786fb-tmp\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.456187 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.456106    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-serving-cert\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.456187 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.456141    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-trusted-ca-bundle\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.456250 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.456221    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-service-ca\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.456283 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.456274    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-config\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.556697 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.556624    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44lp\" (UniqueName: \"kubernetes.io/projected/434cbe69-1f14-42f2-bd9c-446dd3a786fb-kube-api-access-d44lp\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.556697 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.556658    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-oauth-serving-cert\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.556697 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.556683    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-oauth-config\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.556697 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.556699    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/434cbe69-1f14-42f2-bd9c-446dd3a786fb-tmp\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.556998 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.556716    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-serving-cert\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.556998 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.556740    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-trusted-ca-bundle\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.556998 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.556794    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-service-ca\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.556998 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.556842    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-config\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.557204 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.557142    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/434cbe69-1f14-42f2-bd9c-446dd3a786fb-tmp\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.557549 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.557525    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-oauth-serving-cert\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.557659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.557554    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-config\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.557659 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.557635    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-service-ca\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.557782 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.557721    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/434cbe69-1f14-42f2-bd9c-446dd3a786fb-trusted-ca-bundle\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.559280 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.559249    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-oauth-config\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.559372 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.559345    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/434cbe69-1f14-42f2-bd9c-446dd3a786fb-console-serving-cert\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.573717 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.573696    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44lp\" (UniqueName: \"kubernetes.io/projected/434cbe69-1f14-42f2-bd9c-446dd3a786fb-kube-api-access-d44lp\") pod \"console-545cf8b9c5-b5tb6\" (UID: \"434cbe69-1f14-42f2-bd9c-446dd3a786fb\") " pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.581570 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.581539    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:33.872165 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:33.872094    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-545cf8b9c5-b5tb6"]
Oct 30 02:43:33.878954 ip-10-0-133-197 kubenswrapper[4334]: W1030 02:43:33.878930    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434cbe69_1f14_42f2_bd9c_446dd3a786fb.slice/crio-0353a18aa616c8d75d0499a95654e1d6c373aad538d4657c4093d68d9027d9e2 WatchSource:0}: Error finding container 0353a18aa616c8d75d0499a95654e1d6c373aad538d4657c4093d68d9027d9e2: Status 404 returned error can't find the container with id 0353a18aa616c8d75d0499a95654e1d6c373aad538d4657c4093d68d9027d9e2
Oct 30 02:43:34.562112 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:34.562080    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545cf8b9c5-b5tb6" event={"ID":"434cbe69-1f14-42f2-bd9c-446dd3a786fb","Type":"ContainerStarted","Data":"067b451874851c5bf834f2038e74253a4bc2ee133b672c7a159ad2f385379854"}
Oct 30 02:43:34.562495 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:34.562125    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545cf8b9c5-b5tb6" event={"ID":"434cbe69-1f14-42f2-bd9c-446dd3a786fb","Type":"ContainerStarted","Data":"0353a18aa616c8d75d0499a95654e1d6c373aad538d4657c4093d68d9027d9e2"}
Oct 30 02:43:34.598341 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:34.598296    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-545cf8b9c5-b5tb6" podStartSLOduration=1.598283392 podStartE2EDuration="1.598283392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:43:34.598239178 +0000 UTC m=+93.604829432" watchObservedRunningTime="2026-10-30 02:43:34.598283392 +0000 UTC m=+93.604873645"
Oct 30 02:43:43.581762 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.581729    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:43.582210 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.581782    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:43.586556 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.586536    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:43.593530 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.593476    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-664cf94d99-psvrh" podUID="cce6040c-4eb2-4df1-81cd-0b18ad2c050b" containerName="console" containerID="cri-o://83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173" gracePeriod=15
Oct 30 02:43:43.597032 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.597010    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-545cf8b9c5-b5tb6"
Oct 30 02:43:43.687649 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.687622    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bcb4cf79f-fqtvl"]
Oct 30 02:43:43.834144 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.834087    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:43:43.934858 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.934828    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-config\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-config\") pod \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") "
Oct 30 02:43:43.934858 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.934861    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-serving-cert\") pod \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") "
Oct 30 02:43:43.935093 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.934881    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-kube-api-access-lgmc4\" (UniqueName: \"kubernetes.io/projected/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-kube-api-access-lgmc4\") pod \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") "
Oct 30 02:43:43.935093 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.934925    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-oauth-config\") pod \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") "
Oct 30 02:43:43.935093 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935025    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-service-ca\") pod \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") "
Oct 30 02:43:43.935093 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935073    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-trusted-ca-bundle\") pod \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") "
Oct 30 02:43:43.935301 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935109    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-tmp\") pod \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") "
Oct 30 02:43:43.935301 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935161    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-oauth-serving-cert\") pod \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\" (UID: \"cce6040c-4eb2-4df1-81cd-0b18ad2c050b\") "
Oct 30 02:43:43.935410 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935339    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-config" pod "cce6040c-4eb2-4df1-81cd-0b18ad2c050b" (UID: "cce6040c-4eb2-4df1-81cd-0b18ad2c050b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:43.935490 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935400    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-service-ca" pod "cce6040c-4eb2-4df1-81cd-0b18ad2c050b" (UID: "cce6040c-4eb2-4df1-81cd-0b18ad2c050b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:43.935490 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935425    4334 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:43.935490 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935473    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-trusted-ca-bundle" pod "cce6040c-4eb2-4df1-81cd-0b18ad2c050b" (UID: "cce6040c-4eb2-4df1-81cd-0b18ad2c050b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:43.935653 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935596    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-tmp" pod "cce6040c-4eb2-4df1-81cd-0b18ad2c050b" (UID: "cce6040c-4eb2-4df1-81cd-0b18ad2c050b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:43:43.935751 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.935730    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-oauth-serving-cert" pod "cce6040c-4eb2-4df1-81cd-0b18ad2c050b" (UID: "cce6040c-4eb2-4df1-81cd-0b18ad2c050b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:43.937187 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.937156    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-oauth-config" pod "cce6040c-4eb2-4df1-81cd-0b18ad2c050b" (UID: "cce6040c-4eb2-4df1-81cd-0b18ad2c050b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:43:43.937262 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.937223    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-serving-cert" pod "cce6040c-4eb2-4df1-81cd-0b18ad2c050b" (UID: "cce6040c-4eb2-4df1-81cd-0b18ad2c050b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:43:43.937302 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:43.937292    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-kube-api-access-lgmc4" pod "cce6040c-4eb2-4df1-81cd-0b18ad2c050b" (UID: "cce6040c-4eb2-4df1-81cd-0b18ad2c050b"). InnerVolumeSpecName "kube-api-access-lgmc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:43:44.035769 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.035746    4334 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-oauth-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:44.035769 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.035766    4334 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-service-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:44.035927 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.035776    4334 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-trusted-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:44.035927 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.035785    4334 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-tmp\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:44.035927 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.035793    4334 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-oauth-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:44.035927 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.035802    4334 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-console-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:44.035927 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.035810    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgmc4\" (UniqueName: \"kubernetes.io/projected/cce6040c-4eb2-4df1-81cd-0b18ad2c050b-kube-api-access-lgmc4\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:43:44.594786 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.594752    4334 generic.go:356] "Generic (PLEG): container finished" podID="cce6040c-4eb2-4df1-81cd-0b18ad2c050b" containerID="83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173" exitCode=0
Oct 30 02:43:44.595234 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.594823    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664cf94d99-psvrh"
Oct 30 02:43:44.595234 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.594834    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664cf94d99-psvrh" event={"ID":"cce6040c-4eb2-4df1-81cd-0b18ad2c050b","Type":"ContainerDied","Data":"83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173"}
Oct 30 02:43:44.595234 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.594884    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664cf94d99-psvrh" event={"ID":"cce6040c-4eb2-4df1-81cd-0b18ad2c050b","Type":"ContainerDied","Data":"1ef3a8231dc40a1ba800a733065c565fbcbb6d36712e36f06e89092f933dfe30"}
Oct 30 02:43:44.595234 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.594906    4334 scope.go:122] "RemoveContainer" containerID="83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173"
Oct 30 02:43:44.603106 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.603090    4334 scope.go:122] "RemoveContainer" containerID="83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173"
Oct 30 02:43:44.603355 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:43:44.603338    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173\": container with ID starting with 83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173 not found: ID does not exist" containerID="83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173"
Oct 30 02:43:44.603402 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.603364    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173"} err="failed to get container status \"83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173\": rpc error: code = NotFound desc = could not find container \"83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173\": container with ID starting with 83515b2ddb73d9153b8a704e2eda3c1d85c1a1df6c58e11b2480fb16377b5173 not found: ID does not exist"
Oct 30 02:43:44.640995 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.640967    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664cf94d99-psvrh"]
Oct 30 02:43:44.652150 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:44.652131    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-664cf94d99-psvrh"]
Oct 30 02:43:45.157536 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:43:45.157505    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce6040c-4eb2-4df1-81cd-0b18ad2c050b" path="/var/lib/kubelet/pods/cce6040c-4eb2-4df1-81cd-0b18ad2c050b/volumes"
Oct 30 02:43:50.977836 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:43:50.977796    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:44:08.710498 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:08.710435    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-bcb4cf79f-fqtvl" podUID="b6575d45-a32f-491e-8c20-56fbf6bd8604" containerName="console" containerID="cri-o://2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10" gracePeriod=15
Oct 30 02:44:08.952668 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:08.952644    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:44:09.024305 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024232    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-config\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-config\") pod \"b6575d45-a32f-491e-8c20-56fbf6bd8604\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") "
Oct 30 02:44:09.024305 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024281    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-service-ca\") pod \"b6575d45-a32f-491e-8c20-56fbf6bd8604\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") "
Oct 30 02:44:09.024488 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024428    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6575d45-a32f-491e-8c20-56fbf6bd8604-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6575d45-a32f-491e-8c20-56fbf6bd8604-tmp\") pod \"b6575d45-a32f-491e-8c20-56fbf6bd8604\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") "
Oct 30 02:44:09.024488 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024477    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-serving-cert\") pod \"b6575d45-a32f-491e-8c20-56fbf6bd8604\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") "
Oct 30 02:44:09.024583 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024520    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6575d45-a32f-491e-8c20-56fbf6bd8604-kube-api-access-vb9dj\" (UniqueName: \"kubernetes.io/projected/b6575d45-a32f-491e-8c20-56fbf6bd8604-kube-api-access-vb9dj\") pod \"b6575d45-a32f-491e-8c20-56fbf6bd8604\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") "
Oct 30 02:44:09.024583 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024570    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-oauth-config\") pod \"b6575d45-a32f-491e-8c20-56fbf6bd8604\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") "
Oct 30 02:44:09.024681 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024609    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-trusted-ca-bundle\") pod \"b6575d45-a32f-491e-8c20-56fbf6bd8604\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") "
Oct 30 02:44:09.024681 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024627    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-config" pod "b6575d45-a32f-491e-8c20-56fbf6bd8604" (UID: "b6575d45-a32f-491e-8c20-56fbf6bd8604"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:44:09.024681 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024645    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-service-ca" pod "b6575d45-a32f-491e-8c20-56fbf6bd8604" (UID: "b6575d45-a32f-491e-8c20-56fbf6bd8604"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:44:09.024822 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024759    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6575d45-a32f-491e-8c20-56fbf6bd8604-tmp" pod "b6575d45-a32f-491e-8c20-56fbf6bd8604" (UID: "b6575d45-a32f-491e-8c20-56fbf6bd8604"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:44:09.024822 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024637    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-oauth-serving-cert\") pod \"b6575d45-a32f-491e-8c20-56fbf6bd8604\" (UID: \"b6575d45-a32f-491e-8c20-56fbf6bd8604\") "
Oct 30 02:44:09.024957 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.024939    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-oauth-serving-cert" pod "b6575d45-a32f-491e-8c20-56fbf6bd8604" (UID: "b6575d45-a32f-491e-8c20-56fbf6bd8604"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:44:09.025056 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.025034    4334 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6575d45-a32f-491e-8c20-56fbf6bd8604-tmp\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:09.025113 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.025063    4334 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-oauth-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:09.025113 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.025065    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-trusted-ca-bundle" pod "b6575d45-a32f-491e-8c20-56fbf6bd8604" (UID: "b6575d45-a32f-491e-8c20-56fbf6bd8604"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:44:09.025113 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.025080    4334 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:09.025113 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.025096    4334 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-service-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:09.026728 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.026694    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-oauth-config" pod "b6575d45-a32f-491e-8c20-56fbf6bd8604" (UID: "b6575d45-a32f-491e-8c20-56fbf6bd8604"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:44:09.026844 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.026772    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-serving-cert" pod "b6575d45-a32f-491e-8c20-56fbf6bd8604" (UID: "b6575d45-a32f-491e-8c20-56fbf6bd8604"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:44:09.026844 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.026783    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6575d45-a32f-491e-8c20-56fbf6bd8604-kube-api-access-vb9dj" pod "b6575d45-a32f-491e-8c20-56fbf6bd8604" (UID: "b6575d45-a32f-491e-8c20-56fbf6bd8604"). InnerVolumeSpecName "kube-api-access-vb9dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:44:09.125557 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.125510    4334 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-oauth-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:09.125557 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.125551    4334 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6575d45-a32f-491e-8c20-56fbf6bd8604-trusted-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:09.125557 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.125560    4334 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6575d45-a32f-491e-8c20-56fbf6bd8604-console-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:09.125557 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.125569    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb9dj\" (UniqueName: \"kubernetes.io/projected/b6575d45-a32f-491e-8c20-56fbf6bd8604-kube-api-access-vb9dj\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:09.685879 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.685841    4334 generic.go:356] "Generic (PLEG): container finished" podID="b6575d45-a32f-491e-8c20-56fbf6bd8604" containerID="2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10" exitCode=0
Oct 30 02:44:09.686053 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.685894    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcb4cf79f-fqtvl"
Oct 30 02:44:09.686053 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.685922    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcb4cf79f-fqtvl" event={"ID":"b6575d45-a32f-491e-8c20-56fbf6bd8604","Type":"ContainerDied","Data":"2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10"}
Oct 30 02:44:09.686053 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.685982    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcb4cf79f-fqtvl" event={"ID":"b6575d45-a32f-491e-8c20-56fbf6bd8604","Type":"ContainerDied","Data":"75d73395e374ce7a1484d067572cc41a1b1b3d304a3d6026f1ea9dba3b2f3941"}
Oct 30 02:44:09.686053 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.686005    4334 scope.go:122] "RemoveContainer" containerID="2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10"
Oct 30 02:44:09.693593 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.693567    4334 scope.go:122] "RemoveContainer" containerID="2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10"
Oct 30 02:44:09.693852 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:44:09.693835    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10\": container with ID starting with 2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10 not found: ID does not exist" containerID="2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10"
Oct 30 02:44:09.693913 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.693866    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10"} err="failed to get container status \"2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10\": rpc error: code = NotFound desc = could not find container \"2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10\": container with ID starting with 2cf212c74d3ea5865b21042379bc92d66f08251652cde40acc64d20fd6703c10 not found: ID does not exist"
Oct 30 02:44:09.732111 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.732088    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bcb4cf79f-fqtvl"]
Oct 30 02:44:09.743046 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:09.743023    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bcb4cf79f-fqtvl"]
Oct 30 02:44:11.157563 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:11.157527    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6575d45-a32f-491e-8c20-56fbf6bd8604" path="/var/lib/kubelet/pods/b6575d45-a32f-491e-8c20-56fbf6bd8604/volumes"
Oct 30 02:44:20.976933 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:44:20.976847    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:44:50.977427 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:44:50.977388    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:44:54.833362 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:54.833327    4334 generic.go:356] "Generic (PLEG): container finished" podID="4f962975-75bf-409b-b0e5-a340780bffa5" containerID="b0226a0455f5abfae7c5b2bef1a4f9632016c8b409c749fd92301b8ba73681f4" exitCode=0
Oct 30 02:44:54.833744 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:54.833400    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zvcd8-dqps8" event={"ID":"4f962975-75bf-409b-b0e5-a340780bffa5","Type":"ContainerDied","Data":"b0226a0455f5abfae7c5b2bef1a4f9632016c8b409c749fd92301b8ba73681f4"}
Oct 30 02:44:55.956896 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:55.956872    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:44:56.044581 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.044531    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f962975-75bf-409b-b0e5-a340780bffa5-kube-api-access-4fwjx\" (UniqueName: \"kubernetes.io/projected/4f962975-75bf-409b-b0e5-a340780bffa5-kube-api-access-4fwjx\") pod \"4f962975-75bf-409b-b0e5-a340780bffa5\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") "
Oct 30 02:44:56.044737 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.044587    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-trusted-ca-bundle\") pod \"4f962975-75bf-409b-b0e5-a340780bffa5\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") "
Oct 30 02:44:56.044737 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.044662    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f962975-75bf-409b-b0e5-a340780bffa5-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f962975-75bf-409b-b0e5-a340780bffa5-archives-path\") pod \"4f962975-75bf-409b-b0e5-a340780bffa5\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") "
Oct 30 02:44:56.044737 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.044697    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-service-ca-bundle\") pod \"4f962975-75bf-409b-b0e5-a340780bffa5\" (UID: \"4f962975-75bf-409b-b0e5-a340780bffa5\") "
Oct 30 02:44:56.045041 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.045017    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-trusted-ca-bundle" pod "4f962975-75bf-409b-b0e5-a340780bffa5" (UID: "4f962975-75bf-409b-b0e5-a340780bffa5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:44:56.045091 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.045017    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f962975-75bf-409b-b0e5-a340780bffa5-archives-path" pod "4f962975-75bf-409b-b0e5-a340780bffa5" (UID: "4f962975-75bf-409b-b0e5-a340780bffa5"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:44:56.045091 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.045053    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-service-ca-bundle" pod "4f962975-75bf-409b-b0e5-a340780bffa5" (UID: "4f962975-75bf-409b-b0e5-a340780bffa5"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:44:56.046687 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.046668    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f962975-75bf-409b-b0e5-a340780bffa5-kube-api-access-4fwjx" pod "4f962975-75bf-409b-b0e5-a340780bffa5" (UID: "4f962975-75bf-409b-b0e5-a340780bffa5"). InnerVolumeSpecName "kube-api-access-4fwjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:44:56.145518 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.145430    4334 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-service-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:56.145518 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.145483    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwjx\" (UniqueName: \"kubernetes.io/projected/4f962975-75bf-409b-b0e5-a340780bffa5-kube-api-access-4fwjx\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:56.145518 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.145494    4334 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f962975-75bf-409b-b0e5-a340780bffa5-trusted-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:56.145518 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.145504    4334 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f962975-75bf-409b-b0e5-a340780bffa5-archives-path\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 02:44:56.841717 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.841694    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zvcd8-dqps8"
Oct 30 02:44:56.841911 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.841684    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zvcd8-dqps8" event={"ID":"4f962975-75bf-409b-b0e5-a340780bffa5","Type":"ContainerDied","Data":"0c7ee2dbbcdb6b2efc39319ee02afe92b99d999d0a87b1f4ea1a7ab0bdee4473"}
Oct 30 02:44:56.841911 ip-10-0-133-197 kubenswrapper[4334]: I1030 02:44:56.841817    4334 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7ee2dbbcdb6b2efc39319ee02afe92b99d999d0a87b1f4ea1a7ab0bdee4473"
Oct 30 02:45:20.976438 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:45:20.976401    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:45:50.977199 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:45:50.977107    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:46:20.977357 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:46:20.977319    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:46:50.977868 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:46:50.977832    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:47:20.977951 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:47:20.977872    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:47:50.977871 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:47:50.977828    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:48:20.977335 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:48:20.977297    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:48:50.977061 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:48:50.976988    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:49:20.977692 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:49:20.977652    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:49:50.977343 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:49:50.977307    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:50:20.977516 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:50:20.977414    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:50:50.976926 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:50:50.976884    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:51:20.977649 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:51:20.977547    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:51:50.976943 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:51:50.976865    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:52:20.976714 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:52:20.976670    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:52:50.977745 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:52:50.977708    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:53:20.977402 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:53:20.977323    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:53:50.977494 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:53:50.977439    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:54:20.977157 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:54:20.977121    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:54:50.977784 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:54:50.977706    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:55:20.977335 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:55:20.977295    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:55:50.978702 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:55:50.978665    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:56:20.980991 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:56:20.980889    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:56:50.977383 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:56:50.977345    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:57:20.976853 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:57:20.976817    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:57:50.976949 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:57:50.976866    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:58:20.977784 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:58:20.977749    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:58:50.977540 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:58:50.977502    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:59:20.977688 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:59:20.977612    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 02:59:50.977279 ip-10-0-133-197 kubenswrapper[4334]: E1030 02:59:50.977244    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:00:20.977054 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:00:20.977022    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:00:50.976612 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:00:50.976534    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:01:20.977017 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:01:20.976979    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:01:50.978006 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:01:50.977967    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:02:20.977673 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:02:20.977597    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:02:50.977669 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:02:50.977634    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:03:20.977013 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:03:20.976973    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:03:50.977376 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:03:50.977300    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:04:20.978498 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:04:20.978461    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:04:50.978156 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:04:50.978118    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:05:20.978044 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:05:20.977958    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:05:50.977710 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:05:50.977671    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:06:20.977619 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:06:20.977582    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:06:50.977342 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:06:50.977307    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:07:20.977333 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:07:20.977300    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:07:50.977777 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:07:50.977743    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:08:20.978176 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:08:20.978098    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:08:38.951048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951014    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"]
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951249    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b6575d45-a32f-491e-8c20-56fbf6bd8604" containerName="console"
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951259    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6575d45-a32f-491e-8c20-56fbf6bd8604" containerName="console"
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951304    4334 cpu_manager.go:414] "containerMap: removing container" podUID="cce6040c-4eb2-4df1-81cd-0b18ad2c050b" containerName="console"
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951310    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce6040c-4eb2-4df1-81cd-0b18ad2c050b" containerName="console"
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951317    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4f962975-75bf-409b-b0e5-a340780bffa5" containerName="insights-gathering"
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951323    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f962975-75bf-409b-b0e5-a340780bffa5" containerName="insights-gathering"
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951371    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce6040c-4eb2-4df1-81cd-0b18ad2c050b" containerName="console"
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951378    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f962975-75bf-409b-b0e5-a340780bffa5" containerName="insights-gathering"
Oct 30 03:08:38.951521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.951387    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6575d45-a32f-491e-8c20-56fbf6bd8604" containerName="console"
Oct 30 03:08:38.954319 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.954299    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:38.980081 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.980059    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 03:08:38.980081 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.980070    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8pbjr\""
Oct 30 03:08:38.980230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.980086    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 03:08:38.983652 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:38.983628    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"]
Oct 30 03:08:39.045773 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.045741    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.045905 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.045777    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nd6k\" (UniqueName: \"kubernetes.io/projected/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-kube-api-access-2nd6k\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.045905 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.045874    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.146672 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.146643    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.146769 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.146676    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nd6k\" (UniqueName: \"kubernetes.io/projected/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-kube-api-access-2nd6k\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.146769 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.146708    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.146996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.146975    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.147053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.147008    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.165593 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.165569    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nd6k\" (UniqueName: \"kubernetes.io/projected/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-kube-api-access-2nd6k\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.268638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.268580    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:39.560696 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.560613    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"]
Oct 30 03:08:39.566652 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:08:39.566622    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9883ea_9942_4ccd_93b2_b2a8a261fa94.slice/crio-cbefec358cc978059e15536619239b854ff2fea05882bbe7a9c92ad72bb1811a WatchSource:0}: Error finding container cbefec358cc978059e15536619239b854ff2fea05882bbe7a9c92ad72bb1811a: Status 404 returned error can't find the container with id cbefec358cc978059e15536619239b854ff2fea05882bbe7a9c92ad72bb1811a
Oct 30 03:08:39.569066 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:39.569051    4334 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:08:40.390475 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:40.390421    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7" event={"ID":"2b9883ea-9942-4ccd-93b2-b2a8a261fa94","Type":"ContainerStarted","Data":"cbefec358cc978059e15536619239b854ff2fea05882bbe7a9c92ad72bb1811a"}
Oct 30 03:08:45.409616 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:45.409582    4334 generic.go:356] "Generic (PLEG): container finished" podID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerID="f89f2d4ce93b5435c2a95b5386fb3ee7c74905387cfbe33af6122ada884e115a" exitCode=0
Oct 30 03:08:45.410101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:45.409623    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7" event={"ID":"2b9883ea-9942-4ccd-93b2-b2a8a261fa94","Type":"ContainerDied","Data":"f89f2d4ce93b5435c2a95b5386fb3ee7c74905387cfbe33af6122ada884e115a"}
Oct 30 03:08:47.418741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:47.418713    4334 generic.go:356] "Generic (PLEG): container finished" podID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerID="1820c0af5a06d6a216a6257da5f10293b461dd26da2af54b9a3f18e93d0d627e" exitCode=0
Oct 30 03:08:47.419129 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:47.418750    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7" event={"ID":"2b9883ea-9942-4ccd-93b2-b2a8a261fa94","Type":"ContainerDied","Data":"1820c0af5a06d6a216a6257da5f10293b461dd26da2af54b9a3f18e93d0d627e"}
Oct 30 03:08:50.985070 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:08:50.985032    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:08:53.443613 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:53.443585    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7" event={"ID":"2b9883ea-9942-4ccd-93b2-b2a8a261fa94","Type":"ContainerStarted","Data":"25658e7b234a515d7dd327bc172689d97170ff14a2943724eba3d5b2a9006cf1"}
Oct 30 03:08:53.482131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:53.482087    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7" podStartSLOduration=2.760041701 podStartE2EDuration="15.482073632s" totalImagesPullingTime="12.722031931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:53.481500704 +0000 UTC m=+1612.488090959" watchObservedRunningTime="2026-10-30 03:08:53.482073632 +0000 UTC m=+1612.488663885"
Oct 30 03:08:54.449155 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:54.449122    4334 generic.go:356] "Generic (PLEG): container finished" podID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerID="25658e7b234a515d7dd327bc172689d97170ff14a2943724eba3d5b2a9006cf1" exitCode=0
Oct 30 03:08:54.449605 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:54.449156    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7" event={"ID":"2b9883ea-9942-4ccd-93b2-b2a8a261fa94","Type":"ContainerDied","Data":"25658e7b234a515d7dd327bc172689d97170ff14a2943724eba3d5b2a9006cf1"}
Oct 30 03:08:55.568257 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.568238    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:55.681044 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.681013    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-util\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-util\") pod \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") "
Oct 30 03:08:55.681188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.681053    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-kube-api-access-2nd6k\" (UniqueName: \"kubernetes.io/projected/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-kube-api-access-2nd6k\") pod \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") "
Oct 30 03:08:55.681188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.681115    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-bundle\") pod \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\" (UID: \"2b9883ea-9942-4ccd-93b2-b2a8a261fa94\") "
Oct 30 03:08:55.681760 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.681738    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-bundle" pod "2b9883ea-9942-4ccd-93b2-b2a8a261fa94" (UID: "2b9883ea-9942-4ccd-93b2-b2a8a261fa94"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:55.683295 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.683269    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-kube-api-access-2nd6k" pod "2b9883ea-9942-4ccd-93b2-b2a8a261fa94" (UID: "2b9883ea-9942-4ccd-93b2-b2a8a261fa94"). InnerVolumeSpecName "kube-api-access-2nd6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:55.685628 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.685607    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-util" pod "2b9883ea-9942-4ccd-93b2-b2a8a261fa94" (UID: "2b9883ea-9942-4ccd-93b2-b2a8a261fa94"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:08:55.781786 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.781736    4334 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-util\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:08:55.781786 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.781757    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nd6k\" (UniqueName: \"kubernetes.io/projected/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-kube-api-access-2nd6k\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:08:55.781786 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:55.781768    4334 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b9883ea-9942-4ccd-93b2-b2a8a261fa94-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:08:56.457678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:56.457648    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7"
Oct 30 03:08:56.457851 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:56.457648    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpnqc7" event={"ID":"2b9883ea-9942-4ccd-93b2-b2a8a261fa94","Type":"ContainerDied","Data":"cbefec358cc978059e15536619239b854ff2fea05882bbe7a9c92ad72bb1811a"}
Oct 30 03:08:56.457851 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:08:56.457770    4334 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbefec358cc978059e15536619239b854ff2fea05882bbe7a9c92ad72bb1811a"
Oct 30 03:09:02.366863 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.366830    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"]
Oct 30 03:09:02.367245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.367059    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerName="extract"
Oct 30 03:09:02.367245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.367070    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerName="extract"
Oct 30 03:09:02.367245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.367089    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerName="pull"
Oct 30 03:09:02.367245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.367094    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerName="pull"
Oct 30 03:09:02.367245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.367102    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerName="util"
Oct 30 03:09:02.367245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.367107    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerName="util"
Oct 30 03:09:02.367245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.367156    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9883ea-9942-4ccd-93b2-b2a8a261fa94" containerName="extract"
Oct 30 03:09:02.369076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.369056    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"
Oct 30 03:09:02.384253 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.384227    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"]
Oct 30 03:09:02.394387 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.394352    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 03:09:02.394387 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.394366    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:09:02.394540 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.394515    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-x774n\""
Oct 30 03:09:02.536790 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.536760    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qzf\" (UniqueName: \"kubernetes.io/projected/9895a528-2b35-47ee-bc35-4e020c6a7bed-kube-api-access-t9qzf\") pod \"cert-manager-operator-controller-manager-65fc875df4-qdkld\" (UID: \"9895a528-2b35-47ee-bc35-4e020c6a7bed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"
Oct 30 03:09:02.536937 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.536814    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9895a528-2b35-47ee-bc35-4e020c6a7bed-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-qdkld\" (UID: \"9895a528-2b35-47ee-bc35-4e020c6a7bed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"
Oct 30 03:09:02.638039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.637991    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9895a528-2b35-47ee-bc35-4e020c6a7bed-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-qdkld\" (UID: \"9895a528-2b35-47ee-bc35-4e020c6a7bed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"
Oct 30 03:09:02.638099 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.638046    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qzf\" (UniqueName: \"kubernetes.io/projected/9895a528-2b35-47ee-bc35-4e020c6a7bed-kube-api-access-t9qzf\") pod \"cert-manager-operator-controller-manager-65fc875df4-qdkld\" (UID: \"9895a528-2b35-47ee-bc35-4e020c6a7bed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"
Oct 30 03:09:02.638363 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.638344    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9895a528-2b35-47ee-bc35-4e020c6a7bed-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-qdkld\" (UID: \"9895a528-2b35-47ee-bc35-4e020c6a7bed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"
Oct 30 03:09:02.657532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.657512    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qzf\" (UniqueName: \"kubernetes.io/projected/9895a528-2b35-47ee-bc35-4e020c6a7bed-kube-api-access-t9qzf\") pod \"cert-manager-operator-controller-manager-65fc875df4-qdkld\" (UID: \"9895a528-2b35-47ee-bc35-4e020c6a7bed\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"
Oct 30 03:09:02.679337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:02.679317    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"
Oct 30 03:09:03.007477 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:03.007430    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld"]
Oct 30 03:09:03.012109 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:09:03.012081    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9895a528_2b35_47ee_bc35_4e020c6a7bed.slice/crio-a42ca0e7afd054024314bfe9e4d78684491f099b53c54f39ed655ab0a3013dcc WatchSource:0}: Error finding container a42ca0e7afd054024314bfe9e4d78684491f099b53c54f39ed655ab0a3013dcc: Status 404 returned error can't find the container with id a42ca0e7afd054024314bfe9e4d78684491f099b53c54f39ed655ab0a3013dcc
Oct 30 03:09:03.481837 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:03.481802    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld" event={"ID":"9895a528-2b35-47ee-bc35-4e020c6a7bed","Type":"ContainerStarted","Data":"a42ca0e7afd054024314bfe9e4d78684491f099b53c54f39ed655ab0a3013dcc"}
Oct 30 03:09:04.487545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:04.487505    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld" event={"ID":"9895a528-2b35-47ee-bc35-4e020c6a7bed","Type":"ContainerStarted","Data":"9942cd6cf3cbd9bded7ca50450eeb5b1fa25137995206376af2ad124dad2b307"}
Oct 30 03:09:08.967659 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:08.967603    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-qdkld" podStartSLOduration=5.687758824 podStartE2EDuration="6.967587282s" totalImagesPullingTime="1.279828458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:04.564731389 +0000 UTC m=+1623.571321644" watchObservedRunningTime="2026-10-30 03:09:08.967587282 +0000 UTC m=+1627.974177536"
Oct 30 03:09:08.968059 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:08.967938    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"]
Oct 30 03:09:08.971043 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:08.971026    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"
Oct 30 03:09:08.977338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:08.977315    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7690811f-ac85-4311-b173-351db4c06919-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-npl85\" (UID: \"7690811f-ac85-4311-b173-351db4c06919\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"
Oct 30 03:09:08.977425 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:08.977349    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtdt\" (UniqueName: \"kubernetes.io/projected/7690811f-ac85-4311-b173-351db4c06919-kube-api-access-pmtdt\") pod \"cert-manager-cainjector-5c57bbd4fc-npl85\" (UID: \"7690811f-ac85-4311-b173-351db4c06919\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"
Oct 30 03:09:08.994334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:08.994311    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"]
Oct 30 03:09:09.000166 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.000149    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 03:09:09.000272 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.000181    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dtwdp\""
Oct 30 03:09:09.000272 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.000233    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 03:09:09.077686 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.077656    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7690811f-ac85-4311-b173-351db4c06919-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-npl85\" (UID: \"7690811f-ac85-4311-b173-351db4c06919\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"
Oct 30 03:09:09.077841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.077699    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtdt\" (UniqueName: \"kubernetes.io/projected/7690811f-ac85-4311-b173-351db4c06919-kube-api-access-pmtdt\") pod \"cert-manager-cainjector-5c57bbd4fc-npl85\" (UID: \"7690811f-ac85-4311-b173-351db4c06919\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"
Oct 30 03:09:09.099922 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.099897    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtdt\" (UniqueName: \"kubernetes.io/projected/7690811f-ac85-4311-b173-351db4c06919-kube-api-access-pmtdt\") pod \"cert-manager-cainjector-5c57bbd4fc-npl85\" (UID: \"7690811f-ac85-4311-b173-351db4c06919\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"
Oct 30 03:09:09.101501 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.101474    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7690811f-ac85-4311-b173-351db4c06919-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-npl85\" (UID: \"7690811f-ac85-4311-b173-351db4c06919\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"
Oct 30 03:09:09.299337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.299256    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"
Oct 30 03:09:09.600174 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:09.600100    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85"]
Oct 30 03:09:09.605049 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:09:09.605022    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7690811f_ac85_4311_b173_351db4c06919.slice/crio-07c70886a6b905917083b052b864fd0c50dc7fda8172062763cef0d385726e1b WatchSource:0}: Error finding container 07c70886a6b905917083b052b864fd0c50dc7fda8172062763cef0d385726e1b: Status 404 returned error can't find the container with id 07c70886a6b905917083b052b864fd0c50dc7fda8172062763cef0d385726e1b
Oct 30 03:09:10.510708 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:10.510676    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85" event={"ID":"7690811f-ac85-4311-b173-351db4c06919","Type":"ContainerStarted","Data":"07c70886a6b905917083b052b864fd0c50dc7fda8172062763cef0d385726e1b"}
Oct 30 03:09:12.520561 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:12.520525    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85" event={"ID":"7690811f-ac85-4311-b173-351db4c06919","Type":"ContainerStarted","Data":"c67ff6a2788bb90b50d9b81dbf8199d068f529fae32e6386582d8833209e4e60"}
Oct 30 03:09:12.558024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:09:12.557975    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-npl85" podStartSLOduration=2.193144805 podStartE2EDuration="4.557960622s" totalImagesPullingTime="2.364815817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:12.557230361 +0000 UTC m=+1631.563820614" watchObservedRunningTime="2026-10-30 03:09:12.557960622 +0000 UTC m=+1631.564550873"
Oct 30 03:09:20.978340 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:09:20.978300    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:09:50.977906 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:09:50.977871    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:10:20.977435 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:10:20.977398    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:10:50.977866 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:10:50.977818    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:11:09.283950 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.283918    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"]
Oct 30 03:11:09.287381 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.287360    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.305103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.305082    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"]
Oct 30 03:11:09.310734 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.310702    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.310859 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.310758    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2xx\" (UniqueName: \"kubernetes.io/projected/25f7da9e-606e-44cd-bfce-cc515f276952-kube-api-access-2m2xx\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.310933 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.310885    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.315199 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.315182    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 03:11:09.315289 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.315189    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 03:11:09.315339 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.315287    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"07f0b9f07fd02dc1509b226b57404b9c198c3c-8adea440-dockercfg-gmd69\""
Oct 30 03:11:09.377183 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.377150    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"]
Oct 30 03:11:09.380436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.380416    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.398424 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.398403    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"]
Oct 30 03:11:09.404305 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.404284    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e1df189bb31fb8c419c1e848482afda4dccb51-08dd003f-dockercfg-vf5v6\""
Oct 30 03:11:09.411621 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.411597    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.411715 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.411650    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.411774 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.411709    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2xx\" (UniqueName: \"kubernetes.io/projected/25f7da9e-606e-44cd-bfce-cc515f276952-kube-api-access-2m2xx\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.411774 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.411740    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqzl\" (UniqueName: \"kubernetes.io/projected/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-kube-api-access-jcqzl\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.411774 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.411771    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.411932 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.411786    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.412083 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.412065    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.412138 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.412093    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.429834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.429814    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2xx\" (UniqueName: \"kubernetes.io/projected/25f7da9e-606e-44cd-bfce-cc515f276952-kube-api-access-2m2xx\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.512682 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.512641    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.512682 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.512693    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.512905 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.512767    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqzl\" (UniqueName: \"kubernetes.io/projected/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-kube-api-access-jcqzl\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.513111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.513089    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.513147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.513112    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.531238 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.531212    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqzl\" (UniqueName: \"kubernetes.io/projected/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-kube-api-access-jcqzl\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.603472 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.603369    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:09.695604 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.695570    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:09.897090 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.897010    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"]
Oct 30 03:11:09.901911 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:11:09.901870    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f7da9e_606e_44cd_bfce_cc515f276952.slice/crio-c91a5eb1d6ed34a5ccf4a4c28191a6844aa95b000ec09858e71991bc12ac6f98 WatchSource:0}: Error finding container c91a5eb1d6ed34a5ccf4a4c28191a6844aa95b000ec09858e71991bc12ac6f98: Status 404 returned error can't find the container with id c91a5eb1d6ed34a5ccf4a4c28191a6844aa95b000ec09858e71991bc12ac6f98
Oct 30 03:11:09.946157 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.946125    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9" event={"ID":"25f7da9e-606e-44cd-bfce-cc515f276952","Type":"ContainerStarted","Data":"c91a5eb1d6ed34a5ccf4a4c28191a6844aa95b000ec09858e71991bc12ac6f98"}
Oct 30 03:11:09.989499 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:09.989471    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"]
Oct 30 03:11:10.050281 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:11:10.050252    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3033e1_3b89_4530_a2b7_aef0ae6fe649.slice/crio-1d74df8778d56146282e85a77d5e1500c40bae22098b22025837246c4533c3f7 WatchSource:0}: Error finding container 1d74df8778d56146282e85a77d5e1500c40bae22098b22025837246c4533c3f7: Status 404 returned error can't find the container with id 1d74df8778d56146282e85a77d5e1500c40bae22098b22025837246c4533c3f7
Oct 30 03:11:10.952163 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:10.952131    4334 generic.go:356] "Generic (PLEG): container finished" podID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerID="8c11473c156fa1fd3da71e3a3f8cd5e039e2309bd5db5b65a29c8d1daf8855c7" exitCode=0
Oct 30 03:11:10.952624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:10.952212    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n" event={"ID":"ff3033e1-3b89-4530-a2b7-aef0ae6fe649","Type":"ContainerDied","Data":"8c11473c156fa1fd3da71e3a3f8cd5e039e2309bd5db5b65a29c8d1daf8855c7"}
Oct 30 03:11:10.952624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:10.952275    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n" event={"ID":"ff3033e1-3b89-4530-a2b7-aef0ae6fe649","Type":"ContainerStarted","Data":"1d74df8778d56146282e85a77d5e1500c40bae22098b22025837246c4533c3f7"}
Oct 30 03:11:10.953774 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:10.953751    4334 generic.go:356] "Generic (PLEG): container finished" podID="25f7da9e-606e-44cd-bfce-cc515f276952" containerID="e28890db1e1a7101dac7e75643e41e52ca7184dbcdc406a3c79db10a8e6407bc" exitCode=0
Oct 30 03:11:10.953864 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:10.953820    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9" event={"ID":"25f7da9e-606e-44cd-bfce-cc515f276952","Type":"ContainerDied","Data":"e28890db1e1a7101dac7e75643e41e52ca7184dbcdc406a3c79db10a8e6407bc"}
Oct 30 03:11:11.285972 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.285877    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"]
Oct 30 03:11:11.289359 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.289335    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.303321 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.303301    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6fb1a13d619f6ee73e221b33f4898cc831d48b-0ad9862f-dockercfg-nwkr2\""
Oct 30 03:11:11.311542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.311519    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"]
Oct 30 03:11:11.325535 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.325504    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.325770 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.325744    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.325869 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.325833    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcrt\" (UniqueName: \"kubernetes.io/projected/69eba783-b4bc-4238-8b47-a91ab01e3181-kube-api-access-qxcrt\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.426308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.426275    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcrt\" (UniqueName: \"kubernetes.io/projected/69eba783-b4bc-4238-8b47-a91ab01e3181-kube-api-access-qxcrt\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.426481 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.426318    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.426481 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.426345    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.426683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.426665    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.426749 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.426707    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.444683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.444658    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcrt\" (UniqueName: \"kubernetes.io/projected/69eba783-b4bc-4238-8b47-a91ab01e3181-kube-api-access-qxcrt\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.604965 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.604938    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:11.899344 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.899260    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"]
Oct 30 03:11:11.903615 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:11:11.903585    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69eba783_b4bc_4238_8b47_a91ab01e3181.slice/crio-0c12caeaeb9a98bf690d87a31e330bbbfdac3476340aff2effe9f13f835874dd WatchSource:0}: Error finding container 0c12caeaeb9a98bf690d87a31e330bbbfdac3476340aff2effe9f13f835874dd: Status 404 returned error can't find the container with id 0c12caeaeb9a98bf690d87a31e330bbbfdac3476340aff2effe9f13f835874dd
Oct 30 03:11:11.960759 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.960737    4334 generic.go:356] "Generic (PLEG): container finished" podID="25f7da9e-606e-44cd-bfce-cc515f276952" containerID="643548a5311abd5982dcbf7c09a8c5aef1d1e97750677e82f76ef770fca15754" exitCode=0
Oct 30 03:11:11.961123 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.960797    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9" event={"ID":"25f7da9e-606e-44cd-bfce-cc515f276952","Type":"ContainerDied","Data":"643548a5311abd5982dcbf7c09a8c5aef1d1e97750677e82f76ef770fca15754"}
Oct 30 03:11:11.962501 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.962476    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l" event={"ID":"69eba783-b4bc-4238-8b47-a91ab01e3181","Type":"ContainerStarted","Data":"8c679f952877a40eb351c218bea1cbdc3f81e312003c44f3880232d07c42933a"}
Oct 30 03:11:11.962597 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.962524    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l" event={"ID":"69eba783-b4bc-4238-8b47-a91ab01e3181","Type":"ContainerStarted","Data":"0c12caeaeb9a98bf690d87a31e330bbbfdac3476340aff2effe9f13f835874dd"}
Oct 30 03:11:11.964301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.964282    4334 generic.go:356] "Generic (PLEG): container finished" podID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerID="5ae31c068864f234f93e8a557b1e6a7072acbbf7101daf74c1812544641dc3a6" exitCode=0
Oct 30 03:11:11.964383 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:11.964312    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n" event={"ID":"ff3033e1-3b89-4530-a2b7-aef0ae6fe649","Type":"ContainerDied","Data":"5ae31c068864f234f93e8a557b1e6a7072acbbf7101daf74c1812544641dc3a6"}
Oct 30 03:11:12.972482 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:12.972390    4334 generic.go:356] "Generic (PLEG): container finished" podID="25f7da9e-606e-44cd-bfce-cc515f276952" containerID="ddb074cf23b119d6548d2781941d13d84041360fa7c427d9548d2da69ecc3676" exitCode=0
Oct 30 03:11:12.972912 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:12.972487    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9" event={"ID":"25f7da9e-606e-44cd-bfce-cc515f276952","Type":"ContainerDied","Data":"ddb074cf23b119d6548d2781941d13d84041360fa7c427d9548d2da69ecc3676"}
Oct 30 03:11:12.973916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:12.973894    4334 generic.go:356] "Generic (PLEG): container finished" podID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerID="8c679f952877a40eb351c218bea1cbdc3f81e312003c44f3880232d07c42933a" exitCode=0
Oct 30 03:11:12.974005 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:12.973962    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l" event={"ID":"69eba783-b4bc-4238-8b47-a91ab01e3181","Type":"ContainerDied","Data":"8c679f952877a40eb351c218bea1cbdc3f81e312003c44f3880232d07c42933a"}
Oct 30 03:11:12.978399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:12.978378    4334 generic.go:356] "Generic (PLEG): container finished" podID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerID="fd70ea8049ff4f26dff3e3df58744d33b8535ef0f0a3a9095df542d0e92f3c23" exitCode=0
Oct 30 03:11:12.978522 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:12.978406    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n" event={"ID":"ff3033e1-3b89-4530-a2b7-aef0ae6fe649","Type":"ContainerDied","Data":"fd70ea8049ff4f26dff3e3df58744d33b8535ef0f0a3a9095df542d0e92f3c23"}
Oct 30 03:11:13.984667 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:13.984627    4334 generic.go:356] "Generic (PLEG): container finished" podID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerID="693a715a9f5b889659e6b55c85d5b48fa4f5834b59f9391726fe47ff5c6e1226" exitCode=0
Oct 30 03:11:13.985124 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:13.984668    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l" event={"ID":"69eba783-b4bc-4238-8b47-a91ab01e3181","Type":"ContainerDied","Data":"693a715a9f5b889659e6b55c85d5b48fa4f5834b59f9391726fe47ff5c6e1226"}
Oct 30 03:11:14.123866 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.123841    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:14.129169 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.129149    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:14.146618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.146590    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-util\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-util\") pod \"25f7da9e-606e-44cd-bfce-cc515f276952\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") "
Oct 30 03:11:14.146734 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.146652    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-kube-api-access-jcqzl\" (UniqueName: \"kubernetes.io/projected/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-kube-api-access-jcqzl\") pod \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") "
Oct 30 03:11:14.146734 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.146698    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-util\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-util\") pod \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") "
Oct 30 03:11:14.146833 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.146736    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f7da9e-606e-44cd-bfce-cc515f276952-kube-api-access-2m2xx\" (UniqueName: \"kubernetes.io/projected/25f7da9e-606e-44cd-bfce-cc515f276952-kube-api-access-2m2xx\") pod \"25f7da9e-606e-44cd-bfce-cc515f276952\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") "
Oct 30 03:11:14.146833 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.146770    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-bundle\") pod \"25f7da9e-606e-44cd-bfce-cc515f276952\" (UID: \"25f7da9e-606e-44cd-bfce-cc515f276952\") "
Oct 30 03:11:14.146935 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.146833    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-bundle\") pod \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\" (UID: \"ff3033e1-3b89-4530-a2b7-aef0ae6fe649\") "
Oct 30 03:11:14.147493 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.147435    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-bundle" pod "25f7da9e-606e-44cd-bfce-cc515f276952" (UID: "25f7da9e-606e-44cd-bfce-cc515f276952"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:14.147684 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.147659    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-bundle" pod "ff3033e1-3b89-4530-a2b7-aef0ae6fe649" (UID: "ff3033e1-3b89-4530-a2b7-aef0ae6fe649"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:14.148902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.148875    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-kube-api-access-jcqzl" pod "ff3033e1-3b89-4530-a2b7-aef0ae6fe649" (UID: "ff3033e1-3b89-4530-a2b7-aef0ae6fe649"). InnerVolumeSpecName "kube-api-access-jcqzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:14.149278 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.149219    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f7da9e-606e-44cd-bfce-cc515f276952-kube-api-access-2m2xx" pod "25f7da9e-606e-44cd-bfce-cc515f276952" (UID: "25f7da9e-606e-44cd-bfce-cc515f276952"). InnerVolumeSpecName "kube-api-access-2m2xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:14.152756 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.152734    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-util" pod "25f7da9e-606e-44cd-bfce-cc515f276952" (UID: "25f7da9e-606e-44cd-bfce-cc515f276952"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:14.153138 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.153115    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-util" pod "ff3033e1-3b89-4530-a2b7-aef0ae6fe649" (UID: "ff3033e1-3b89-4530-a2b7-aef0ae6fe649"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:14.247663 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.247630    4334 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:14.247663 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.247659    4334 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-util\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:14.247863 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.247672    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcqzl\" (UniqueName: \"kubernetes.io/projected/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-kube-api-access-jcqzl\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:14.247863 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.247686    4334 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff3033e1-3b89-4530-a2b7-aef0ae6fe649-util\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:14.247863 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.247701    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2xx\" (UniqueName: \"kubernetes.io/projected/25f7da9e-606e-44cd-bfce-cc515f276952-kube-api-access-2m2xx\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:14.247863 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.247712    4334 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f7da9e-606e-44cd-bfce-cc515f276952-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:14.990898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.990861    4334 generic.go:356] "Generic (PLEG): container finished" podID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerID="c5a6d4a431cd1ae57d3b5411fc6ef5b1f2e24106e2af061625dcbb9b5c784112" exitCode=0
Oct 30 03:11:14.991344 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.990950    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l" event={"ID":"69eba783-b4bc-4238-8b47-a91ab01e3181","Type":"ContainerDied","Data":"c5a6d4a431cd1ae57d3b5411fc6ef5b1f2e24106e2af061625dcbb9b5c784112"}
Oct 30 03:11:14.992683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.992662    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n"
Oct 30 03:11:14.992785 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.992665    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974rtq7n" event={"ID":"ff3033e1-3b89-4530-a2b7-aef0ae6fe649","Type":"ContainerDied","Data":"1d74df8778d56146282e85a77d5e1500c40bae22098b22025837246c4533c3f7"}
Oct 30 03:11:14.992785 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.992770    4334 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d74df8778d56146282e85a77d5e1500c40bae22098b22025837246c4533c3f7"
Oct 30 03:11:14.994675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.994656    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9"
Oct 30 03:11:14.994675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.994667    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054mxqj9" event={"ID":"25f7da9e-606e-44cd-bfce-cc515f276952","Type":"ContainerDied","Data":"c91a5eb1d6ed34a5ccf4a4c28191a6844aa95b000ec09858e71991bc12ac6f98"}
Oct 30 03:11:14.994800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:14.994703    4334 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c91a5eb1d6ed34a5ccf4a4c28191a6844aa95b000ec09858e71991bc12ac6f98"
Oct 30 03:11:16.115031 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.115007    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:16.160879 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.160852    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-util\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-util\") pod \"69eba783-b4bc-4238-8b47-a91ab01e3181\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") "
Oct 30 03:11:16.161046 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.160918    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69eba783-b4bc-4238-8b47-a91ab01e3181-kube-api-access-qxcrt\" (UniqueName: \"kubernetes.io/projected/69eba783-b4bc-4238-8b47-a91ab01e3181-kube-api-access-qxcrt\") pod \"69eba783-b4bc-4238-8b47-a91ab01e3181\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") "
Oct 30 03:11:16.161046 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.160959    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-bundle\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-bundle\") pod \"69eba783-b4bc-4238-8b47-a91ab01e3181\" (UID: \"69eba783-b4bc-4238-8b47-a91ab01e3181\") "
Oct 30 03:11:16.162429 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.162397    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-bundle" pod "69eba783-b4bc-4238-8b47-a91ab01e3181" (UID: "69eba783-b4bc-4238-8b47-a91ab01e3181"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:16.163118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.163091    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69eba783-b4bc-4238-8b47-a91ab01e3181-kube-api-access-qxcrt" pod "69eba783-b4bc-4238-8b47-a91ab01e3181" (UID: "69eba783-b4bc-4238-8b47-a91ab01e3181"). InnerVolumeSpecName "kube-api-access-qxcrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:16.166164 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.166144    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-util" pod "69eba783-b4bc-4238-8b47-a91ab01e3181" (UID: "69eba783-b4bc-4238-8b47-a91ab01e3181"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:11:16.262048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.261968    4334 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-util\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:16.262048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.261993    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxcrt\" (UniqueName: \"kubernetes.io/projected/69eba783-b4bc-4238-8b47-a91ab01e3181-kube-api-access-qxcrt\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:16.262048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:16.262004    4334 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69eba783-b4bc-4238-8b47-a91ab01e3181-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:11:17.005217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:17.005178    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l" event={"ID":"69eba783-b4bc-4238-8b47-a91ab01e3181","Type":"ContainerDied","Data":"0c12caeaeb9a98bf690d87a31e330bbbfdac3476340aff2effe9f13f835874dd"}
Oct 30 03:11:17.005217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:17.005209    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfcvj6l"
Oct 30 03:11:17.005217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:17.005222    4334 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c12caeaeb9a98bf690d87a31e330bbbfdac3476340aff2effe9f13f835874dd"
Oct 30 03:11:20.121528 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121496    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"]
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121749    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerName="extract"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121759    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerName="extract"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121773    4334 cpu_manager.go:414] "containerMap: removing container" podUID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerName="pull"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121777    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerName="pull"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121811    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerName="util"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121817    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerName="util"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121823    4334 cpu_manager.go:414] "containerMap: removing container" podUID="25f7da9e-606e-44cd-bfce-cc515f276952" containerName="pull"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121829    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f7da9e-606e-44cd-bfce-cc515f276952" containerName="pull"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121837    4334 cpu_manager.go:414] "containerMap: removing container" podUID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerName="extract"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121841    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerName="extract"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121848    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerName="pull"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121852    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerName="pull"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121858    4334 cpu_manager.go:414] "containerMap: removing container" podUID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerName="util"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121862    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerName="util"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121876    4334 cpu_manager.go:414] "containerMap: removing container" podUID="25f7da9e-606e-44cd-bfce-cc515f276952" containerName="util"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121884    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f7da9e-606e-44cd-bfce-cc515f276952" containerName="util"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121893    4334 cpu_manager.go:414] "containerMap: removing container" podUID="25f7da9e-606e-44cd-bfce-cc515f276952" containerName="extract"
Oct 30 03:11:20.121898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121897    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f7da9e-606e-44cd-bfce-cc515f276952" containerName="extract"
Oct 30 03:11:20.122513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121941    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="69eba783-b4bc-4238-8b47-a91ab01e3181" containerName="extract"
Oct 30 03:11:20.122513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121951    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3033e1-3b89-4530-a2b7-aef0ae6fe649" containerName="extract"
Oct 30 03:11:20.122513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.121958    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f7da9e-606e-44cd-bfce-cc515f276952" containerName="extract"
Oct 30 03:11:20.124771 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.124755    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:20.152034 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.151990    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\""
Oct 30 03:11:20.154563 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.154539    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-j5t2z\""
Oct 30 03:11:20.155802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.155781    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"]
Oct 30 03:11:20.157092 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.157074    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:11:20.157228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.157098    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-metrics\""
Oct 30 03:11:20.188788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.188767    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96dfd055-afe2-4a32-962f-c6d04e9c0ee7-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4\" (UID: \"96dfd055-afe2-4a32-962f-c6d04e9c0ee7\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:20.188890 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.188796    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rr5t\" (UniqueName: \"kubernetes.io/projected/96dfd055-afe2-4a32-962f-c6d04e9c0ee7-kube-api-access-2rr5t\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4\" (UID: \"96dfd055-afe2-4a32-962f-c6d04e9c0ee7\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:20.289742 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.289696    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96dfd055-afe2-4a32-962f-c6d04e9c0ee7-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4\" (UID: \"96dfd055-afe2-4a32-962f-c6d04e9c0ee7\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:20.289883 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.289787    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rr5t\" (UniqueName: \"kubernetes.io/projected/96dfd055-afe2-4a32-962f-c6d04e9c0ee7-kube-api-access-2rr5t\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4\" (UID: \"96dfd055-afe2-4a32-962f-c6d04e9c0ee7\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:20.292146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.292120    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96dfd055-afe2-4a32-962f-c6d04e9c0ee7-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4\" (UID: \"96dfd055-afe2-4a32-962f-c6d04e9c0ee7\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:20.317563 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.317535    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rr5t\" (UniqueName: \"kubernetes.io/projected/96dfd055-afe2-4a32-962f-c6d04e9c0ee7-kube-api-access-2rr5t\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4\" (UID: \"96dfd055-afe2-4a32-962f-c6d04e9c0ee7\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:20.436200 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.436172    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:20.757510 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.757407    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"]
Oct 30 03:11:20.762050 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:11:20.762024    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96dfd055_afe2_4a32_962f_c6d04e9c0ee7.slice/crio-7eeac3f817771396f25dbd5878dc80fa25306dd3c4ba127d14188bc75a63057a WatchSource:0}: Error finding container 7eeac3f817771396f25dbd5878dc80fa25306dd3c4ba127d14188bc75a63057a: Status 404 returned error can't find the container with id 7eeac3f817771396f25dbd5878dc80fa25306dd3c4ba127d14188bc75a63057a
Oct 30 03:11:20.813348 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.813324    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"]
Oct 30 03:11:20.817728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.817711    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:20.846245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.846222    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\""
Oct 30 03:11:20.859204 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.859182    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"]
Oct 30 03:11:20.893968 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.893943    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b38dcab6-a007-4ff9-a343-76e7b22690d5-webhook-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:20.894084 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.894002    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b38dcab6-a007-4ff9-a343-76e7b22690d5-apiservice-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:20.894084 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.894062    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2zl\" (UniqueName: \"kubernetes.io/projected/b38dcab6-a007-4ff9-a343-76e7b22690d5-kube-api-access-ws2zl\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:20.978142 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:11:20.978105    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:11:20.994553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.994531    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b38dcab6-a007-4ff9-a343-76e7b22690d5-webhook-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:20.994670 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.994573    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b38dcab6-a007-4ff9-a343-76e7b22690d5-apiservice-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:20.994670 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.994598    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2zl\" (UniqueName: \"kubernetes.io/projected/b38dcab6-a007-4ff9-a343-76e7b22690d5-kube-api-access-ws2zl\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:20.996870 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.996846    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b38dcab6-a007-4ff9-a343-76e7b22690d5-apiservice-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:20.996959 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:20.996874    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b38dcab6-a007-4ff9-a343-76e7b22690d5-webhook-cert\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:21.014055 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:21.014004    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2zl\" (UniqueName: \"kubernetes.io/projected/b38dcab6-a007-4ff9-a343-76e7b22690d5-kube-api-access-ws2zl\") pod \"opentelemetry-operator-webhook-55c856bfb8-82kms\" (UID: \"b38dcab6-a007-4ff9-a343-76e7b22690d5\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:21.024495 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:21.024473    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4" event={"ID":"96dfd055-afe2-4a32-962f-c6d04e9c0ee7","Type":"ContainerStarted","Data":"7eeac3f817771396f25dbd5878dc80fa25306dd3c4ba127d14188bc75a63057a"}
Oct 30 03:11:21.129223 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:21.129191    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:21.466834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:21.466789    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"]
Oct 30 03:11:21.471073 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:11:21.471046    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38dcab6_a007_4ff9_a343_76e7b22690d5.slice/crio-392ff42f91bd85df6224f02168b64502e160fb9e35c6953fbf8fbe5b2ca992bc WatchSource:0}: Error finding container 392ff42f91bd85df6224f02168b64502e160fb9e35c6953fbf8fbe5b2ca992bc: Status 404 returned error can't find the container with id 392ff42f91bd85df6224f02168b64502e160fb9e35c6953fbf8fbe5b2ca992bc
Oct 30 03:11:22.031738 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:22.031699    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms" event={"ID":"b38dcab6-a007-4ff9-a343-76e7b22690d5","Type":"ContainerStarted","Data":"392ff42f91bd85df6224f02168b64502e160fb9e35c6953fbf8fbe5b2ca992bc"}
Oct 30 03:11:23.037812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:23.037776    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4" event={"ID":"96dfd055-afe2-4a32-962f-c6d04e9c0ee7","Type":"ContainerStarted","Data":"a89babd0fdb838004be12d9d02db97f177d546a7d18581b0984f7fd85622863a"}
Oct 30 03:11:23.038284 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:23.037869    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:23.039521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:23.039502    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms" event={"ID":"b38dcab6-a007-4ff9-a343-76e7b22690d5","Type":"ContainerStarted","Data":"54779ed07de993a2b6f94ba4ae6ae4197f844af12d96731a309d1aa0464113cd"}
Oct 30 03:11:23.039589 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:23.039539    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:23.111135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:23.111087    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4" podStartSLOduration=1.509813144 podStartE2EDuration="3.111076286s" totalImagesPullingTime="1.601263142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:23.110495725 +0000 UTC m=+1762.117085980" watchObservedRunningTime="2026-10-30 03:11:23.111076286 +0000 UTC m=+1762.117666539"
Oct 30 03:11:23.156828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:23.156779    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms" podStartSLOduration=2.068156753 podStartE2EDuration="3.156765352s" totalImagesPullingTime="1.088608599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:23.155723791 +0000 UTC m=+1762.162314046" watchObservedRunningTime="2026-10-30 03:11:23.156765352 +0000 UTC m=+1762.163355596"
Oct 30 03:11:34.046349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.046316    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-v8zt4"
Oct 30 03:11:34.047800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.047782    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-55c856bfb8-82kms"
Oct 30 03:11:34.752909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.752876    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v"]
Oct 30 03:11:34.756192 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.756175    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v"
Oct 30 03:11:34.790789 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.790764    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:11:34.790990 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.790961    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\""
Oct 30 03:11:34.791203 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.791191    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-dockercfg-m4ksp\""
Oct 30 03:11:34.807541 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.807519    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtw2\" (UniqueName: \"kubernetes.io/projected/63684e83-2888-4e73-a4e6-ad4cf19e0e9b-kube-api-access-kjtw2\") pod \"obo-prometheus-operator-6ddb664c4f-9sc4v\" (UID: \"63684e83-2888-4e73-a4e6-ad4cf19e0e9b\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v"
Oct 30 03:11:34.817528 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.817507    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v"]
Oct 30 03:11:34.908261 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.908231    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtw2\" (UniqueName: \"kubernetes.io/projected/63684e83-2888-4e73-a4e6-ad4cf19e0e9b-kube-api-access-kjtw2\") pod \"obo-prometheus-operator-6ddb664c4f-9sc4v\" (UID: \"63684e83-2888-4e73-a4e6-ad4cf19e0e9b\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v"
Oct 30 03:11:34.931817 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:34.931787    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtw2\" (UniqueName: \"kubernetes.io/projected/63684e83-2888-4e73-a4e6-ad4cf19e0e9b-kube-api-access-kjtw2\") pod \"obo-prometheus-operator-6ddb664c4f-9sc4v\" (UID: \"63684e83-2888-4e73-a4e6-ad4cf19e0e9b\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v"
Oct 30 03:11:35.024337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.024276    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"]
Oct 30 03:11:35.027323 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.027304    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"
Oct 30 03:11:35.045043 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.045021    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\""
Oct 30 03:11:35.047727 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.047707    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-lrnv7\""
Oct 30 03:11:35.049662 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.049641    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"]
Oct 30 03:11:35.068974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.068958    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v"
Oct 30 03:11:35.109755 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.109721    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8523c5e8-8219-409e-917f-b56aabcfa4ac-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d\" (UID: \"8523c5e8-8219-409e-917f-b56aabcfa4ac\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"
Oct 30 03:11:35.109875 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.109838    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8523c5e8-8219-409e-917f-b56aabcfa4ac-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d\" (UID: \"8523c5e8-8219-409e-917f-b56aabcfa4ac\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"
Oct 30 03:11:35.210190 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.210153    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8523c5e8-8219-409e-917f-b56aabcfa4ac-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d\" (UID: \"8523c5e8-8219-409e-917f-b56aabcfa4ac\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"
Oct 30 03:11:35.210371 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.210208    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8523c5e8-8219-409e-917f-b56aabcfa4ac-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d\" (UID: \"8523c5e8-8219-409e-917f-b56aabcfa4ac\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"
Oct 30 03:11:35.212602 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.212568    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8523c5e8-8219-409e-917f-b56aabcfa4ac-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d\" (UID: \"8523c5e8-8219-409e-917f-b56aabcfa4ac\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"
Oct 30 03:11:35.212602 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.212596    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8523c5e8-8219-409e-917f-b56aabcfa4ac-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d\" (UID: \"8523c5e8-8219-409e-917f-b56aabcfa4ac\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"
Oct 30 03:11:35.339881 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.339812    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"
Oct 30 03:11:35.393748 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.393714    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v"]
Oct 30 03:11:35.398671 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:11:35.398633    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63684e83_2888_4e73_a4e6_ad4cf19e0e9b.slice/crio-ba196bde0992c966f2b0948e01cdc90220f2ccab82e6b879ea59866f40b1ecd8 WatchSource:0}: Error finding container ba196bde0992c966f2b0948e01cdc90220f2ccab82e6b879ea59866f40b1ecd8: Status 404 returned error can't find the container with id ba196bde0992c966f2b0948e01cdc90220f2ccab82e6b879ea59866f40b1ecd8
Oct 30 03:11:35.484587 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.484561    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"]
Oct 30 03:11:35.488975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.488954    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:35.506160 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.506138    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"observability-operator-sa-dockercfg-wvx5t\""
Oct 30 03:11:35.506281 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.506253    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"observability-operator-tls\""
Oct 30 03:11:35.519299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.519279    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"]
Oct 30 03:11:35.612990 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.612914    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnrr\" (UniqueName: \"kubernetes.io/projected/6759fd87-35d2-4228-9018-1e72abcae3ae-kube-api-access-wfnrr\") pod \"observability-operator-599c6f89d-svg64\" (UID: \"6759fd87-35d2-4228-9018-1e72abcae3ae\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:35.612990 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.612977    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/6759fd87-35d2-4228-9018-1e72abcae3ae-observability-operator-tls\") pod \"observability-operator-599c6f89d-svg64\" (UID: \"6759fd87-35d2-4228-9018-1e72abcae3ae\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:35.668705 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.668675    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d"]
Oct 30 03:11:35.672977 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:11:35.672949    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8523c5e8_8219_409e_917f_b56aabcfa4ac.slice/crio-3c52b9b3d4a22fb7fbff8e21f90b0481940e82820bba3fbcdccdc546a298bd78 WatchSource:0}: Error finding container 3c52b9b3d4a22fb7fbff8e21f90b0481940e82820bba3fbcdccdc546a298bd78: Status 404 returned error can't find the container with id 3c52b9b3d4a22fb7fbff8e21f90b0481940e82820bba3fbcdccdc546a298bd78
Oct 30 03:11:35.714180 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.714153    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/6759fd87-35d2-4228-9018-1e72abcae3ae-observability-operator-tls\") pod \"observability-operator-599c6f89d-svg64\" (UID: \"6759fd87-35d2-4228-9018-1e72abcae3ae\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:35.714277 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.714203    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnrr\" (UniqueName: \"kubernetes.io/projected/6759fd87-35d2-4228-9018-1e72abcae3ae-kube-api-access-wfnrr\") pod \"observability-operator-599c6f89d-svg64\" (UID: \"6759fd87-35d2-4228-9018-1e72abcae3ae\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:35.716576 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.716560    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/6759fd87-35d2-4228-9018-1e72abcae3ae-observability-operator-tls\") pod \"observability-operator-599c6f89d-svg64\" (UID: \"6759fd87-35d2-4228-9018-1e72abcae3ae\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:35.736203 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.736182    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnrr\" (UniqueName: \"kubernetes.io/projected/6759fd87-35d2-4228-9018-1e72abcae3ae-kube-api-access-wfnrr\") pod \"observability-operator-599c6f89d-svg64\" (UID: \"6759fd87-35d2-4228-9018-1e72abcae3ae\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:35.801125 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:35.801102    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:36.099264 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:36.099225    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v" event={"ID":"63684e83-2888-4e73-a4e6-ad4cf19e0e9b","Type":"ContainerStarted","Data":"ba196bde0992c966f2b0948e01cdc90220f2ccab82e6b879ea59866f40b1ecd8"}
Oct 30 03:11:36.101011 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:36.100986    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d" event={"ID":"8523c5e8-8219-409e-917f-b56aabcfa4ac","Type":"ContainerStarted","Data":"3c52b9b3d4a22fb7fbff8e21f90b0481940e82820bba3fbcdccdc546a298bd78"}
Oct 30 03:11:36.122128 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:36.122068    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"]
Oct 30 03:11:36.129511 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:11:36.129483    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6759fd87_35d2_4228_9018_1e72abcae3ae.slice/crio-8740dd1fa979b4ddcff6405f65a17c3070a3591e06f2992c21fc06fbcb2f0056 WatchSource:0}: Error finding container 8740dd1fa979b4ddcff6405f65a17c3070a3591e06f2992c21fc06fbcb2f0056: Status 404 returned error can't find the container with id 8740dd1fa979b4ddcff6405f65a17c3070a3591e06f2992c21fc06fbcb2f0056
Oct 30 03:11:37.109163 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:37.109125    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64" event={"ID":"6759fd87-35d2-4228-9018-1e72abcae3ae","Type":"ContainerStarted","Data":"8740dd1fa979b4ddcff6405f65a17c3070a3591e06f2992c21fc06fbcb2f0056"}
Oct 30 03:11:40.123268 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:40.123230    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d" event={"ID":"8523c5e8-8219-409e-917f-b56aabcfa4ac","Type":"ContainerStarted","Data":"24d5a09e16924998046319060f0dff06999bd48bff6f0f13ebaff6b3177d9788"}
Oct 30 03:11:40.124965 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:40.124931    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v" event={"ID":"63684e83-2888-4e73-a4e6-ad4cf19e0e9b","Type":"ContainerStarted","Data":"80f915d11f614d598c0a877c7cda46777d63c1b11b12264084049f37b6e9869d"}
Oct 30 03:11:40.192867 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:40.192817    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-79b8ff74fb-kgr2d" podStartSLOduration=2.694192702 podStartE2EDuration="6.19280319s" totalImagesPullingTime="3.498610488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:40.186096464 +0000 UTC m=+1779.192686718" watchObservedRunningTime="2026-10-30 03:11:40.19280319 +0000 UTC m=+1779.199393444"
Oct 30 03:11:40.228721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:40.228674    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9sc4v" podStartSLOduration=2.454905349 podStartE2EDuration="6.228660015s" totalImagesPullingTime="3.773754666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:40.227347987 +0000 UTC m=+1779.233938241" watchObservedRunningTime="2026-10-30 03:11:40.228660015 +0000 UTC m=+1779.235250265"
Oct 30 03:11:41.132303 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:41.132265    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64" event={"ID":"6759fd87-35d2-4228-9018-1e72abcae3ae","Type":"ContainerStarted","Data":"9d4d9ad707c58db9e5125ee50829894f68b5ac988fa93243eb718b348541b47b"}
Oct 30 03:11:41.132830 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:41.132646    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:41.134138 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:41.134106    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64"
Oct 30 03:11:41.251279 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:11:41.251220    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-svg64" podStartSLOduration=2.300808624 podStartE2EDuration="6.251202301s" totalImagesPullingTime="3.950393677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:41.204097118 +0000 UTC m=+1780.210687371" watchObservedRunningTime="2026-10-30 03:11:41.251202301 +0000 UTC m=+1780.257792556"
Oct 30 03:11:50.977944 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:11:50.977902    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:12:00.606953 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.606908    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"]
Oct 30 03:12:00.612898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.612878    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.636074 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.636049    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"d28b81ce69fab5e58d2cd73462dde4041bbc20-487d35f4-dockercfg-lqtfm\""
Oct 30 03:12:00.647518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.647489    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 03:12:00.647518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.647515    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 03:12:00.662505 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.662484    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"]
Oct 30 03:12:00.719591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.719559    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28twh\" (UniqueName: \"kubernetes.io/projected/76e9637d-e037-492d-ac80-6b22804d0d46-kube-api-access-28twh\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.719720 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.719595    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.719720 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.719689    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.820880 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.820846    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.821028 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.820915    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28twh\" (UniqueName: \"kubernetes.io/projected/76e9637d-e037-492d-ac80-6b22804d0d46-kube-api-access-28twh\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.821028 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.820936    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.821261 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.821246    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.821327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.821264    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.846674 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.846647    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28twh\" (UniqueName: \"kubernetes.io/projected/76e9637d-e037-492d-ac80-6b22804d0d46-kube-api-access-28twh\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:00.930356 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:00.930330    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:01.249994 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:01.249922    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"]
Oct 30 03:12:01.252781 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:12:01.252747    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e9637d_e037_492d_ac80_6b22804d0d46.slice/crio-90a0212e7ab06d20e19972deb1f49f2631916a25f8562ba986ff3c95be25df0d WatchSource:0}: Error finding container 90a0212e7ab06d20e19972deb1f49f2631916a25f8562ba986ff3c95be25df0d: Status 404 returned error can't find the container with id 90a0212e7ab06d20e19972deb1f49f2631916a25f8562ba986ff3c95be25df0d
Oct 30 03:12:02.230548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:02.230515    4334 generic.go:356] "Generic (PLEG): container finished" podID="76e9637d-e037-492d-ac80-6b22804d0d46" containerID="a6648563701836a4b8a4ee5e1e9c28e3434b27141cd9d3b4b175161a20812228" exitCode=0
Oct 30 03:12:02.230991 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:02.230570    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5" event={"ID":"76e9637d-e037-492d-ac80-6b22804d0d46","Type":"ContainerDied","Data":"a6648563701836a4b8a4ee5e1e9c28e3434b27141cd9d3b4b175161a20812228"}
Oct 30 03:12:02.230991 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:02.230614    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5" event={"ID":"76e9637d-e037-492d-ac80-6b22804d0d46","Type":"ContainerStarted","Data":"90a0212e7ab06d20e19972deb1f49f2631916a25f8562ba986ff3c95be25df0d"}
Oct 30 03:12:03.236913 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:03.236879    4334 generic.go:356] "Generic (PLEG): container finished" podID="76e9637d-e037-492d-ac80-6b22804d0d46" containerID="31627459cfb453a913f3744c89b50066fc06c0cb219841e9caaac2dbba890bc6" exitCode=0
Oct 30 03:12:03.237309 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:03.236941    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5" event={"ID":"76e9637d-e037-492d-ac80-6b22804d0d46","Type":"ContainerDied","Data":"31627459cfb453a913f3744c89b50066fc06c0cb219841e9caaac2dbba890bc6"}
Oct 30 03:12:04.243530 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:04.243490    4334 generic.go:356] "Generic (PLEG): container finished" podID="76e9637d-e037-492d-ac80-6b22804d0d46" containerID="e096da19fdd5f618856bd8cf08e21c3e612b6e17006e4d3244b4273f4900f748" exitCode=0
Oct 30 03:12:04.243983 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:04.243550    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5" event={"ID":"76e9637d-e037-492d-ac80-6b22804d0d46","Type":"ContainerDied","Data":"e096da19fdd5f618856bd8cf08e21c3e612b6e17006e4d3244b4273f4900f748"}
Oct 30 03:12:05.371606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.371585    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:05.460351 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.460321    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-util\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-util\") pod \"76e9637d-e037-492d-ac80-6b22804d0d46\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") "
Oct 30 03:12:05.460543 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.460364    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-bundle\") pod \"76e9637d-e037-492d-ac80-6b22804d0d46\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") "
Oct 30 03:12:05.460543 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.460401    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e9637d-e037-492d-ac80-6b22804d0d46-kube-api-access-28twh\" (UniqueName: \"kubernetes.io/projected/76e9637d-e037-492d-ac80-6b22804d0d46-kube-api-access-28twh\") pod \"76e9637d-e037-492d-ac80-6b22804d0d46\" (UID: \"76e9637d-e037-492d-ac80-6b22804d0d46\") "
Oct 30 03:12:05.460954 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.460926    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-bundle" pod "76e9637d-e037-492d-ac80-6b22804d0d46" (UID: "76e9637d-e037-492d-ac80-6b22804d0d46"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:05.462688 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.462667    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e9637d-e037-492d-ac80-6b22804d0d46-kube-api-access-28twh" pod "76e9637d-e037-492d-ac80-6b22804d0d46" (UID: "76e9637d-e037-492d-ac80-6b22804d0d46"). InnerVolumeSpecName "kube-api-access-28twh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:05.466425 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.466392    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-util" pod "76e9637d-e037-492d-ac80-6b22804d0d46" (UID: "76e9637d-e037-492d-ac80-6b22804d0d46"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:05.561227 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.561171    4334 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-util\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:12:05.561227 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.561192    4334 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76e9637d-e037-492d-ac80-6b22804d0d46-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:12:05.561227 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:05.561203    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28twh\" (UniqueName: \"kubernetes.io/projected/76e9637d-e037-492d-ac80-6b22804d0d46-kube-api-access-28twh\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:12:06.254413 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:06.254377    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5" event={"ID":"76e9637d-e037-492d-ac80-6b22804d0d46","Type":"ContainerDied","Data":"90a0212e7ab06d20e19972deb1f49f2631916a25f8562ba986ff3c95be25df0d"}
Oct 30 03:12:06.254413 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:06.254402    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaad9hch5"
Oct 30 03:12:06.254634 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:06.254422    4334 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a0212e7ab06d20e19972deb1f49f2631916a25f8562ba986ff3c95be25df0d"
Oct 30 03:12:20.978310 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:20.978198    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:12:39.301265 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.301229    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:12:39.301772 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.301690    4334 cpu_manager.go:414] "containerMap: removing container" podUID="76e9637d-e037-492d-ac80-6b22804d0d46" containerName="util"
Oct 30 03:12:39.301772 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.301710    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e9637d-e037-492d-ac80-6b22804d0d46" containerName="util"
Oct 30 03:12:39.301772 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.301725    4334 cpu_manager.go:414] "containerMap: removing container" podUID="76e9637d-e037-492d-ac80-6b22804d0d46" containerName="pull"
Oct 30 03:12:39.301772 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.301733    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e9637d-e037-492d-ac80-6b22804d0d46" containerName="pull"
Oct 30 03:12:39.301772 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.301746    4334 cpu_manager.go:414] "containerMap: removing container" podUID="76e9637d-e037-492d-ac80-6b22804d0d46" containerName="extract"
Oct 30 03:12:39.301772 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.301755    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e9637d-e037-492d-ac80-6b22804d0d46" containerName="extract"
Oct 30 03:12:39.302069 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.301849    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e9637d-e037-492d-ac80-6b22804d0d46" containerName="extract"
Oct 30 03:12:39.304416 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.304399    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.326878 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.326855    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:12:39.335587 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.335566    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:12:39.335587 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.335584    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:12:39.335754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.335584    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:12:39.335754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.335640    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:12:39.335754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.335699    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:12:39.403015 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.402989    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.403179 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.403045    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.403179 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.403155    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl65s\" (UniqueName: \"kubernetes.io/projected/8fbebb56-1bce-44ce-a996-830e401855da-kube-api-access-xl65s\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.403284 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.403192    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8fbebb56-1bce-44ce-a996-830e401855da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.504009 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.503975    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8fbebb56-1bce-44ce-a996-830e401855da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.504184 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.504021    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.504184 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.504057    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.504184 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.504084    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl65s\" (UniqueName: \"kubernetes.io/projected/8fbebb56-1bce-44ce-a996-830e401855da-kube-api-access-xl65s\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.504692 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.504653    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8fbebb56-1bce-44ce-a996-830e401855da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.506589 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.506567    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.506684 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.506623    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.523052 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.523032    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl65s\" (UniqueName: \"kubernetes.io/projected/8fbebb56-1bce-44ce-a996-830e401855da-kube-api-access-xl65s\") pod \"data-science-collector-collector-0\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.615106 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.615023    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:39.980050 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:39.980017    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:12:39.984303 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:12:39.984275    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbebb56_1bce_44ce_a996_830e401855da.slice/crio-03245c37459a7b2b3b0c907c5343d9f389fb6fec57f4be4e995e3c7798849fa6 WatchSource:0}: Error finding container 03245c37459a7b2b3b0c907c5343d9f389fb6fec57f4be4e995e3c7798849fa6: Status 404 returned error can't find the container with id 03245c37459a7b2b3b0c907c5343d9f389fb6fec57f4be4e995e3c7798849fa6
Oct 30 03:12:40.407556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:40.407525    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8fbebb56-1bce-44ce-a996-830e401855da","Type":"ContainerStarted","Data":"03245c37459a7b2b3b0c907c5343d9f389fb6fec57f4be4e995e3c7798849fa6"}
Oct 30 03:12:43.723204 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:43.723175    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:12:43.723470 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:43.723243    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:12:44.057002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.056929    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:12:44.060648 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.060627    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.074885 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.074862    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:12:44.085611 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.085571    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:12:44.085611 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.085593    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:12:44.085765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.085616    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:12:44.085765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.085620    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:12:44.085765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.085621    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:12:44.085765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.085657    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:12:44.085765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.085657    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-bkrdk\""
Oct 30 03:12:44.087224 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.087207    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:12:44.143318 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.143290    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.143479 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.143322    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.143479 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.143355    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.143591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.143492    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.143591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.143532    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.143591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.143560    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.143745 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.143672    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.143745 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.143716    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhff\" (UniqueName: \"kubernetes.io/projected/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-kube-api-access-wqhff\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.244885 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.244860    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.244906    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245096 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245074    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245148 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245115    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhff\" (UniqueName: \"kubernetes.io/projected/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-kube-api-access-wqhff\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245148 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245140    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245257 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245161    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245257 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245205    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245257 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245241    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245852 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245825    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245927    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.245995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.245990    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.246119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.246007    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.247420 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.247400    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.247840 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.247819    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.247993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.247961    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.263056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.263033    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhff\" (UniqueName: \"kubernetes.io/projected/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-kube-api-access-wqhff\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:44.427741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.427710    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8fbebb56-1bce-44ce-a996-830e401855da","Type":"ContainerStarted","Data":"b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b"}
Oct 30 03:12:44.466995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:44.466939    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.7306665049999999 podStartE2EDuration="5.46691984s" totalImagesPullingTime="3.736253335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:44.46586637 +0000 UTC m=+1843.472456625" watchObservedRunningTime="2026-10-30 03:12:44.46691984 +0000 UTC m=+1843.473510094"
Oct 30 03:12:48.578837 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:48.578805    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:48.605246 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:48.605221    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:48.679634 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:48.679606    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:48.679741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:48.679715    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:48.692755 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:48.692737    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:48.875500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:48.875440    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:48.898666 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:48.898642    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:49.176498 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:49.176470    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:49.510389 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:49.510295    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:12:49.516415 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:12:49.516389    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab51b3c6_dd6c_4916_9c64_7221f2f83c4c.slice/crio-d54d0d773deb632842bc73ab2e7fff4f6fbdb6ca01636127c9e09f88ec8606aa WatchSource:0}: Error finding container d54d0d773deb632842bc73ab2e7fff4f6fbdb6ca01636127c9e09f88ec8606aa: Status 404 returned error can't find the container with id d54d0d773deb632842bc73ab2e7fff4f6fbdb6ca01636127c9e09f88ec8606aa
Oct 30 03:12:50.456801 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:50.456762    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c","Type":"ContainerStarted","Data":"d54d0d773deb632842bc73ab2e7fff4f6fbdb6ca01636127c9e09f88ec8606aa"}
Oct 30 03:12:50.978465 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:50.978419    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:12:53.474016 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:53.473946    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c","Type":"ContainerStarted","Data":"13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e"}
Oct 30 03:12:55.486733 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:55.486691    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c","Type":"ContainerStarted","Data":"3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6"}
Oct 30 03:12:56.493886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:56.493803    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c","Type":"ContainerStarted","Data":"1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1"}
Oct 30 03:12:56.494320 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:56.493993    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:56.494320 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:56.494046    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:56.496577 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:56.496556    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:56.502406 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:56.502389    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:56.559079 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:56.559030    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=9.013609819 podStartE2EDuration="15.559017836s" totalImagesPullingTime="6.545408017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:12:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:56.558324346 +0000 UTC m=+1855.564914599" watchObservedRunningTime="2026-10-30 03:12:56.559017836 +0000 UTC m=+1855.565608089"
Oct 30 03:12:57.498227 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:57.498197    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:12:57.845194 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:57.845096    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:12:57.845359 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:57.845337    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="8fbebb56-1bce-44ce-a996-830e401855da" containerName="otc-container" containerID="cri-o://b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b" gracePeriod=30
Oct 30 03:12:58.091291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.091266    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:58.253933 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.253896    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbebb56-1bce-44ce-a996-830e401855da-kube-api-access-xl65s\" (UniqueName: \"kubernetes.io/projected/8fbebb56-1bce-44ce-a996-830e401855da-kube-api-access-xl65s\") pod \"8fbebb56-1bce-44ce-a996-830e401855da\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") "
Oct 30 03:12:58.254126 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.253974    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbebb56-1bce-44ce-a996-830e401855da-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8fbebb56-1bce-44ce-a996-830e401855da-otc-internal\") pod \"8fbebb56-1bce-44ce-a996-830e401855da\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") "
Oct 30 03:12:58.254126 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.254030    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-otlp-tls-certs\") pod \"8fbebb56-1bce-44ce-a996-830e401855da\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") "
Oct 30 03:12:58.254126 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.254119    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-monitoring-tls-certs\") pod \"8fbebb56-1bce-44ce-a996-830e401855da\" (UID: \"8fbebb56-1bce-44ce-a996-830e401855da\") "
Oct 30 03:12:58.254426 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.254398    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbebb56-1bce-44ce-a996-830e401855da-otc-internal" pod "8fbebb56-1bce-44ce-a996-830e401855da" (UID: "8fbebb56-1bce-44ce-a996-830e401855da"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:58.256289 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.256252    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-otlp-tls-certs" pod "8fbebb56-1bce-44ce-a996-830e401855da" (UID: "8fbebb56-1bce-44ce-a996-830e401855da"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:58.256390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.256287    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-monitoring-tls-certs" pod "8fbebb56-1bce-44ce-a996-830e401855da" (UID: "8fbebb56-1bce-44ce-a996-830e401855da"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:58.256390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.256333    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbebb56-1bce-44ce-a996-830e401855da-kube-api-access-xl65s" pod "8fbebb56-1bce-44ce-a996-830e401855da" (UID: "8fbebb56-1bce-44ce-a996-830e401855da"). InnerVolumeSpecName "kube-api-access-xl65s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:58.313069 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.313034    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:12:58.354993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.354932    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.354993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.354966    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl65s\" (UniqueName: \"kubernetes.io/projected/8fbebb56-1bce-44ce-a996-830e401855da-kube-api-access-xl65s\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.354993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.354983    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8fbebb56-1bce-44ce-a996-830e401855da-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.354993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.354995    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbebb56-1bce-44ce-a996-830e401855da-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:12:58.503994 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.503916    4334 generic.go:356] "Generic (PLEG): container finished" podID="8fbebb56-1bce-44ce-a996-830e401855da" containerID="b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b" exitCode=0
Oct 30 03:12:58.503994 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.503953    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8fbebb56-1bce-44ce-a996-830e401855da","Type":"ContainerDied","Data":"b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b"}
Oct 30 03:12:58.504467 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.504014    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8fbebb56-1bce-44ce-a996-830e401855da","Type":"ContainerDied","Data":"03245c37459a7b2b3b0c907c5343d9f389fb6fec57f4be4e995e3c7798849fa6"}
Oct 30 03:12:58.504467 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.504040    4334 scope.go:122] "RemoveContainer" containerID="b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b"
Oct 30 03:12:58.504467 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.503966    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:12:58.504467 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.504260    4334 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" secret="" err="secret \"tempo-data-science-tempomonolithic-dockercfg-bkrdk\" not found"
Oct 30 03:12:58.513647 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.513631    4334 scope.go:122] "RemoveContainer" containerID="b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b"
Oct 30 03:12:58.513917 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.513900    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b\": container with ID starting with b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b not found: ID does not exist" containerID="b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b"
Oct 30 03:12:58.513968 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.513925    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b"} err="failed to get container status \"b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b\": rpc error: code = NotFound desc = could not find container \"b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b\": container with ID starting with b83936a627fc9c2ddcb0ed42549ff83edcb4303d02f237020eb6f17d5b82057b not found: ID does not exist"
Oct 30 03:12:58.550824 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.550796    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:12:58.553986 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:58.553964    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:12:58.657858 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.657833    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:12:58.657858 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.657849    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:12:58.657996 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.657913    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:12:58.657996 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.657836    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:12:58.657996 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.657928    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:12:58.657996 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.657913    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:12:59.157890634 +0000 UTC m=+1858.164480885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:12:58.657996 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.657989    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:12:59.157979998 +0000 UTC m=+1858.164570235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:12:58.658162 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.657999    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:12:59.157994108 +0000 UTC m=+1858.164584340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:12:58.658162 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.658009    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:12:59.158004354 +0000 UTC m=+1858.164594586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:12:58.658162 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.658068    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:12:58.658162 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.658079    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:12:59.158063933 +0000 UTC m=+1858.164654169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:12:58.658162 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.658105    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:12:59.158098285 +0000 UTC m=+1858.164688517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:12:58.658162 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.658129    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:12:58.658360 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:58.658194    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:12:59.158184256 +0000 UTC m=+1858.164774512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:12:59.159352 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:59.159323    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbebb56-1bce-44ce-a996-830e401855da" path="/var/lib/kubelet/pods/8fbebb56-1bce-44ce-a996-830e401855da/volumes"
Oct 30 03:12:59.162800 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162771    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:12:59.162800 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162795    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162821    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162836    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:00.162817424 +0000 UTC m=+1859.169407669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162851    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162857    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162867    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:00.162850642 +0000 UTC m=+1859.169440874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162885    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:00.162874959 +0000 UTC m=+1859.169465198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162891    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162917    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162901    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:00.16289188 +0000 UTC m=+1859.169482112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162959    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:00.162943614 +0000 UTC m=+1859.169533858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162979    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:00.162968085 +0000 UTC m=+1859.169558322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:12:59.162988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:12:59.162998    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:00.162990899 +0000 UTC m=+1859.169581132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:12:59.509292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:59.509199    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo" containerID="cri-o://13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e" gracePeriod=30
Oct 30 03:12:59.509292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:59.509223    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo-gateway-opa" containerID="cri-o://1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1" gracePeriod=30
Oct 30 03:12:59.509292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:12:59.509247    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo-gateway" containerID="cri-o://3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6" gracePeriod=30
Oct 30 03:13:00.172042 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172012    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:13:00.172042 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172028    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172068    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172080    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172107    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172081    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:02.172062254 +0000 UTC m=+1861.178652486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172135    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:02.17212117 +0000 UTC m=+1861.178711405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172080    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172171    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172181    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:02.172174469 +0000 UTC m=+1861.178764702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172196    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:02.17218713 +0000 UTC m=+1861.178777363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172209    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:02.172202669 +0000 UTC m=+1861.178792900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172220    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:02.172215246 +0000 UTC m=+1861.178805477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:13:00.172289 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:00.172230    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls podName:ab51b3c6-dd6c-4916-9c64-7221f2f83c4c nodeName:}" failed. No retries permitted until 2026-10-30 03:13:02.172224596 +0000 UTC m=+1861.178814828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:13:00.520848 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.520766    4334 generic.go:356] "Generic (PLEG): container finished" podID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerID="1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1" exitCode=0
Oct 30 03:13:00.520848 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.520791    4334 generic.go:356] "Generic (PLEG): container finished" podID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerID="13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e" exitCode=0
Oct 30 03:13:00.521267 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.520839    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c","Type":"ContainerDied","Data":"1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1"}
Oct 30 03:13:00.521267 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.520930    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c","Type":"ContainerDied","Data":"13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e"}
Oct 30 03:13:00.750170 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.750145    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:13:00.879391 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879317    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879391 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879350    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879391 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879377    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879639 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879416    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879639 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879464    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879639 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879516    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-kube-api-access-wqhff\" (UniqueName: \"kubernetes.io/projected/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-kube-api-access-wqhff\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879796 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879640    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879796 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879684    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879796 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879717    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls\") pod \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\" (UID: \"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c\") "
Oct 30 03:13:00.879796 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879771    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:00.879796 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879789    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:00.880062 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879974    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:00.880062 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.879994    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:00.880182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.880140    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:00.880182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.880153    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:00.881926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.881898    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:00.882058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.881922    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:00.882058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.881938    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:00.882140 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.882110    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-kube-api-access-wqhff" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "kube-api-access-wqhff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:00.883747 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.883730    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" (UID: "ab51b3c6-dd6c-4916-9c64-7221f2f83c4c"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:13:00.981143 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.981117    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:13:00.981143 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.981139    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:00.981331 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.981150    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:00.981331 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.981160    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:00.981331 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.981169    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-gateway-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:00.981331 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.981177    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:00.981331 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.981187    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhff\" (UniqueName: \"kubernetes.io/projected/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c-kube-api-access-wqhff\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:00.989993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:00.989979    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:13:01.081570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.081546    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\""
Oct 30 03:13:01.527716 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.527630    4334 generic.go:356] "Generic (PLEG): container finished" podID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerID="3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6" exitCode=0
Oct 30 03:13:01.527716 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.527698    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:13:01.527716 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.527713    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c","Type":"ContainerDied","Data":"3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6"}
Oct 30 03:13:01.528226 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.527767    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ab51b3c6-dd6c-4916-9c64-7221f2f83c4c","Type":"ContainerDied","Data":"d54d0d773deb632842bc73ab2e7fff4f6fbdb6ca01636127c9e09f88ec8606aa"}
Oct 30 03:13:01.528226 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.527799    4334 scope.go:122] "RemoveContainer" containerID="1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1"
Oct 30 03:13:01.535741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.535715    4334 scope.go:122] "RemoveContainer" containerID="3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6"
Oct 30 03:13:01.543172 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.543155    4334 scope.go:122] "RemoveContainer" containerID="13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e"
Oct 30 03:13:01.550207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.550189    4334 scope.go:122] "RemoveContainer" containerID="1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1"
Oct 30 03:13:01.550482 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:01.550459    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1\": container with ID starting with 1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1 not found: ID does not exist" containerID="1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1"
Oct 30 03:13:01.550548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.550495    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1"} err="failed to get container status \"1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1\": rpc error: code = NotFound desc = could not find container \"1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1\": container with ID starting with 1d1634a4b170fcc311eac9cc5a8006df8192634ccc231a5c24d0d1fee61eb5a1 not found: ID does not exist"
Oct 30 03:13:01.550548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.550516    4334 scope.go:122] "RemoveContainer" containerID="3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6"
Oct 30 03:13:01.550760 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:01.550744    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6\": container with ID starting with 3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6 not found: ID does not exist" containerID="3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6"
Oct 30 03:13:01.550797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.550769    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6"} err="failed to get container status \"3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6\": rpc error: code = NotFound desc = could not find container \"3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6\": container with ID starting with 3c422663164921ca57210b40e386793bbf25ae44e08a14ad7172bf804273a4b6 not found: ID does not exist"
Oct 30 03:13:01.550797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.550785    4334 scope.go:122] "RemoveContainer" containerID="13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e"
Oct 30 03:13:01.551017 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:01.550996    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e\": container with ID starting with 13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e not found: ID does not exist" containerID="13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e"
Oct 30 03:13:01.551056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.551024    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e"} err="failed to get container status \"13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e\": rpc error: code = NotFound desc = could not find container \"13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e\": container with ID starting with 13b51b1188c9dbfec15e538453468a1db6afb65482a7da51192199f76f35358e not found: ID does not exist"
Oct 30 03:13:01.587051 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.587030    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:13:01.598335 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:01.598312    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:13:03.099911 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.099879    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"]
Oct 30 03:13:03.100379 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100357    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo"
Oct 30 03:13:03.100379 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100375    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo"
Oct 30 03:13:03.100525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100391    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo-gateway-opa"
Oct 30 03:13:03.100525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100399    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo-gateway-opa"
Oct 30 03:13:03.100525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100420    4334 cpu_manager.go:414] "containerMap: removing container" podUID="8fbebb56-1bce-44ce-a996-830e401855da" containerName="otc-container"
Oct 30 03:13:03.100525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100429    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbebb56-1bce-44ce-a996-830e401855da" containerName="otc-container"
Oct 30 03:13:03.100525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100440    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo-gateway"
Oct 30 03:13:03.100525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100477    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo-gateway"
Oct 30 03:13:03.100828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100591    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo"
Oct 30 03:13:03.100828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100605    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo-gateway-opa"
Oct 30 03:13:03.100828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100617    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbebb56-1bce-44ce-a996-830e401855da" containerName="otc-container"
Oct 30 03:13:03.100828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.100626    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" containerName="tempo-gateway"
Oct 30 03:13:03.105259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.105239    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.135942 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.135915    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"]
Oct 30 03:13:03.152775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.152744    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"]
Oct 30 03:13:03.156573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.156553    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.164018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.163627    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab51b3c6-dd6c-4916-9c64-7221f2f83c4c" path="/var/lib/kubelet/pods/ab51b3c6-dd6c-4916-9c64-7221f2f83c4c/volumes"
Oct 30 03:13:03.164018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.163731    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 03:13:03.164018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.163766    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 03:13:03.164018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.163808    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:13:03.164018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.163820    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-dz75c\""
Oct 30 03:13:03.164018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.163856    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:13:03.164018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.163891    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 03:13:03.164018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.163903    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:13:03.176380 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.176356    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"]
Oct 30 03:13:03.197354 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.197326    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.197468 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.197400    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.197540 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.197465    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.197588 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.197566    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.197683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.197668    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vjx\" (UniqueName: \"kubernetes.io/projected/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-kube-api-access-n2vjx\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.264159 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.264137    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 03:13:03.264385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.264366    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-cgjh6\""
Oct 30 03:13:03.264508 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.264413    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 03:13:03.298695 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.298674    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.298763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.298700    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cw7\" (UniqueName: \"kubernetes.io/projected/b01a2d62-bcab-408e-acd7-7e875276d0e0-kube-api-access-n4cw7\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.298763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.298735    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b01a2d62-bcab-408e-acd7-7e875276d0e0-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.298763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.298756    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b01a2d62-bcab-408e-acd7-7e875276d0e0-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.298928 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.298907    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vjx\" (UniqueName: \"kubernetes.io/projected/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-kube-api-access-n2vjx\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.298990 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.298946    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.298990 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.298980    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.299091 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.299022    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.299423 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.299394    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.299423 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.299417    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.299577 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.299534    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.301361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.301339    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.318237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.318210    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vjx\" (UniqueName: \"kubernetes.io/projected/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-kube-api-access-n2vjx\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-tcrpm\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.399915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.399857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cw7\" (UniqueName: \"kubernetes.io/projected/b01a2d62-bcab-408e-acd7-7e875276d0e0-kube-api-access-n4cw7\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.399915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.399901    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b01a2d62-bcab-408e-acd7-7e875276d0e0-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.400043 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.399919    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b01a2d62-bcab-408e-acd7-7e875276d0e0-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.400542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.400519    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b01a2d62-bcab-408e-acd7-7e875276d0e0-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.402143 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.402124    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b01a2d62-bcab-408e-acd7-7e875276d0e0-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.419457 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.419431    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cw7\" (UniqueName: \"kubernetes.io/projected/b01a2d62-bcab-408e-acd7-7e875276d0e0-kube-api-access-n4cw7\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-2lwf4\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.420184 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.420166    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:13:03.471877 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.471847    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:13:03.715581 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.715547    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"]
Oct 30 03:13:03.719956 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:13:03.719929    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeb82e9_d212_419e_9cc3_24fcf9c91d33.slice/crio-91dd55f9961b2cf0b2a513ec1b3fbeb233b53e30aca0f0491606193dc84f175f WatchSource:0}: Error finding container 91dd55f9961b2cf0b2a513ec1b3fbeb233b53e30aca0f0491606193dc84f175f: Status 404 returned error can't find the container with id 91dd55f9961b2cf0b2a513ec1b3fbeb233b53e30aca0f0491606193dc84f175f
Oct 30 03:13:03.779982 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:03.779957    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"]
Oct 30 03:13:03.784248 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:13:03.784220    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01a2d62_bcab_408e_acd7_7e875276d0e0.slice/crio-d908fbe91f7b751ff5b5e6ddf119665738144703342443d026be080ff0fd103c WatchSource:0}: Error finding container d908fbe91f7b751ff5b5e6ddf119665738144703342443d026be080ff0fd103c: Status 404 returned error can't find the container with id d908fbe91f7b751ff5b5e6ddf119665738144703342443d026be080ff0fd103c
Oct 30 03:13:04.548747 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.548643    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" event={"ID":"6aeb82e9-d212-419e-9cc3-24fcf9c91d33","Type":"ContainerStarted","Data":"91dd55f9961b2cf0b2a513ec1b3fbeb233b53e30aca0f0491606193dc84f175f"}
Oct 30 03:13:04.550430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.550368    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" event={"ID":"b01a2d62-bcab-408e-acd7-7e875276d0e0","Type":"ContainerStarted","Data":"d908fbe91f7b751ff5b5e6ddf119665738144703342443d026be080ff0fd103c"}
Oct 30 03:13:04.675202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.675165    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:04.679003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.678981    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.697007 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.696982    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:04.721475 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.721433    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:13:04.773485 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.772070    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:13:04.773485 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.772173    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:13:04.773485 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.772211    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:13:04.773485 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.772222    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:13:04.773485 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.772256    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:13:04.812756 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.812682    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqgr\" (UniqueName: \"kubernetes.io/projected/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-kube-api-access-nlqgr\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.812915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.812766    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.812915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.812806    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.812915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.812848    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.812915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.812878    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.812915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.812903    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.914161 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.914126    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.914325 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.914177    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.914325 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.914222    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqgr\" (UniqueName: \"kubernetes.io/projected/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-kube-api-access-nlqgr\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.914325 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.914306    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.914532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.914347    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.914532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.914390    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.915541 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:04.915516    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found
Oct 30 03:13:04.915618 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:04.915604    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert podName:b3513b6a-fcbf-47bd-b6df-fce58e0c4e04 nodeName:}" failed. No retries permitted until 2026-10-30 03:13:05.415572818 +0000 UTC m=+1864.422163074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04") : secret "data-science-collector-ta-client-cert" not found
Oct 30 03:13:04.918385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.917952    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.918385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.918264    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.920059 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.920031    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.928532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.928165    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:04.937952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:04.937919    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqgr\" (UniqueName: \"kubernetes.io/projected/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-kube-api-access-nlqgr\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:05.418760 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:05.418644    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:05.418974 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:05.418807    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found
Oct 30 03:13:05.418974 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:05.418883    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert podName:b3513b6a-fcbf-47bd-b6df-fce58e0c4e04 nodeName:}" failed. No retries permitted until 2026-10-30 03:13:06.418861931 +0000 UTC m=+1865.425452164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04") : secret "data-science-collector-ta-client-cert" not found
Oct 30 03:13:06.428856 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:06.428811    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:06.431898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:06.431863    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:06.503387 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:06.503347    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:06.571093 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:06.571060    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" event={"ID":"6aeb82e9-d212-419e-9cc3-24fcf9c91d33","Type":"ContainerStarted","Data":"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699"}
Oct 30 03:13:06.801759 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:06.801683    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:06.805708 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:13:06.805667    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3513b6a_fcbf_47bd_b6df_fce58e0c4e04.slice/crio-91cc7be09cae6a70c97b6394eff3b9be22cc038b40d8d0b7e5e374ee52e04dc7 WatchSource:0}: Error finding container 91cc7be09cae6a70c97b6394eff3b9be22cc038b40d8d0b7e5e374ee52e04dc7: Status 404 returned error can't find the container with id 91cc7be09cae6a70c97b6394eff3b9be22cc038b40d8d0b7e5e374ee52e04dc7
Oct 30 03:13:06.807967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:06.807933    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:13:06.808069 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:06.808011    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:13:07.578528 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:07.578491    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" event={"ID":"6aeb82e9-d212-419e-9cc3-24fcf9c91d33","Type":"ContainerStarted","Data":"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045"}
Oct 30 03:13:07.580375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:07.580350    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04","Type":"ContainerStarted","Data":"b3bc8f4fd0f7d706bfc451709cc033d02e4869fcdc9ab935a764e895dc292e36"}
Oct 30 03:13:07.580546 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:07.580401    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04","Type":"ContainerStarted","Data":"91cc7be09cae6a70c97b6394eff3b9be22cc038b40d8d0b7e5e374ee52e04dc7"}
Oct 30 03:13:07.581713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:07.581694    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" event={"ID":"b01a2d62-bcab-408e-acd7-7e875276d0e0","Type":"ContainerStarted","Data":"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8"}
Oct 30 03:13:07.616319 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:07.616278    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" podStartSLOduration=0.925901164 podStartE2EDuration="4.616266597s" totalImagesPullingTime="3.690365433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:07.615363755 +0000 UTC m=+1866.621954006" watchObservedRunningTime="2026-10-30 03:13:07.616266597 +0000 UTC m=+1866.622856850"
Oct 30 03:13:07.653369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:07.653318    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=3.653305525 podStartE2EDuration="3.653305525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:07.651540533 +0000 UTC m=+1866.658130790" watchObservedRunningTime="2026-10-30 03:13:07.653305525 +0000 UTC m=+1866.659895779"
Oct 30 03:13:08.226621 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.226583    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:13:08.230608 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.230585    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.260114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.260086    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:13:08.324776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.324748    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 03:13:08.324776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.324758    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 03:13:08.324776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.324758    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 03:13:08.325011 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.324808    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:13:08.325011 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.324999    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-zzqx2\""
Oct 30 03:13:08.345067 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.345040    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.345067 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.345070    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.345240 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.345102    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbc5\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-kube-api-access-zjbc5\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.345240 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.345171    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.345240 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.345211    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.345344 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.345311    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.345344 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.345333    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.445963 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.445905    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.446141 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.446011    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbc5\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-kube-api-access-zjbc5\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.446141 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.446042    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.446141 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.446068    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.446141 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.446122    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.446366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.446156    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.446366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.446224    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.446505 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.446358    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.448939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.448915    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.449056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.449019    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.449155 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.449138    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.449195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.449140    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.449195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.449181    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.464909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.464878    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbc5\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-kube-api-access-zjbc5\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.547875 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.547802    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:08.589239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.589200    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" event={"ID":"b01a2d62-bcab-408e-acd7-7e875276d0e0","Type":"ContainerStarted","Data":"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d"}
Oct 30 03:13:08.629000 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.628940    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" podStartSLOduration=1.963462263 podStartE2EDuration="5.628921597s" totalImagesPullingTime="3.665459334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:08.627322913 +0000 UTC m=+1867.633913168" watchObservedRunningTime="2026-10-30 03:13:08.628921597 +0000 UTC m=+1867.635511853"
Oct 30 03:13:08.861726 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:08.861631    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:13:08.867075 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:13:08.867040    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b95646_94f7_4f0c_b193_b1a530517b68.slice/crio-eb43f5cd71b6d2bf59906531bd8733d833e829e3ec4d40138740eb2e76a58e57 WatchSource:0}: Error finding container eb43f5cd71b6d2bf59906531bd8733d833e829e3ec4d40138740eb2e76a58e57: Status 404 returned error can't find the container with id eb43f5cd71b6d2bf59906531bd8733d833e829e3ec4d40138740eb2e76a58e57
Oct 30 03:13:09.597015 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:09.596974    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"f0b95646-94f7-4f0c-b193-b1a530517b68","Type":"ContainerStarted","Data":"eb43f5cd71b6d2bf59906531bd8733d833e829e3ec4d40138740eb2e76a58e57"}
Oct 30 03:13:10.606118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:10.606082    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"f0b95646-94f7-4f0c-b193-b1a530517b68","Type":"ContainerStarted","Data":"3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371"}
Oct 30 03:13:11.340984 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.340950    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:13:11.345770 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.345747    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.366117 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.366089    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:13:11.415663 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.415639    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:13:11.416601 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.416582    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:13:11.416683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.416593    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:13:11.416683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.416624    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:13:11.416683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.416630    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:13:11.416683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.416650    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-jz5xx\""
Oct 30 03:13:11.417812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.417793    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:13:11.417812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.417806    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:13:11.417916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.417793    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:13:11.417973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.417961    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:13:11.475695 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475671    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.475811 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475700    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.475811 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475719    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.475811 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475799    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfq2\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-kube-api-access-pvfq2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.475911 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475861    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.475911 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475888    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.475975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475917    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.475975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475943    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.476033 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.475974    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.476079 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.476061    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.476124 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.476114    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.476154 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.476143    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577332 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577305    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577443 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577339    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577443 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577356    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577443 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577382    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577615    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577695 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577666    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577749 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577734    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577806 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577764    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577863 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577800    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577863 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577855    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfq2\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-kube-api-access-pvfq2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577891    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.577974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.577927    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.578546 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.578327    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.578546 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.578523    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.578731 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.578568    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.580595 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.580534    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.580595 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.580571    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.580746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.580705    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.581266 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.581241    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.581374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.581352    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.581576 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.581556    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.581657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.581640    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.581723 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.581686    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:11.600370 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:11.600326    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfq2\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-kube-api-access-pvfq2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:14.626199 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:14.626110    4334 generic.go:356] "Generic (PLEG): container finished" podID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerID="3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371" exitCode=0
Oct 30 03:13:14.626199 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:14.626183    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"f0b95646-94f7-4f0c-b193-b1a530517b68","Type":"ContainerDied","Data":"3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371"}
Oct 30 03:13:16.638064 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:16.638034    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"f0b95646-94f7-4f0c-b193-b1a530517b68","Type":"ContainerStarted","Data":"33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7"}
Oct 30 03:13:16.638422 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:16.638092    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"f0b95646-94f7-4f0c-b193-b1a530517b68","Type":"ContainerStarted","Data":"3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4"}
Oct 30 03:13:16.638422 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:16.638373    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:16.704458 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:16.704408    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.68136786 podStartE2EDuration="8.704394418s" totalImagesPullingTime="2.636666108s" totalInitContainerRuntime="3.38636045s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:16.691966087 +0000 UTC m=+1875.698556340" watchObservedRunningTime="2026-10-30 03:13:16.704394418 +0000 UTC m=+1875.710984726"
Oct 30 03:13:19.544070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:19.544040    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:19.570901 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:19.570872    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:19.644443 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:19.644413    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:19.644587 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:19.644558    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:19.658385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:19.658365    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:19.838309 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:19.838281    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:19.861033 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:19.861007    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:20.065607 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:20.065576    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:20.401979 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:13:20.401946    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42d6464_fd67_4e6e_a690_7d1c1ea93cc8.slice/crio-12ba85d94a5eb7fb0aa4428bf95be148db86b96004634ac4528ef1227ab0cc42 WatchSource:0}: Error finding container 12ba85d94a5eb7fb0aa4428bf95be148db86b96004634ac4528ef1227ab0cc42: Status 404 returned error can't find the container with id 12ba85d94a5eb7fb0aa4428bf95be148db86b96004634ac4528ef1227ab0cc42
Oct 30 03:13:20.409677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:20.409656    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:13:20.657319 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:20.657285    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerStarted","Data":"12ba85d94a5eb7fb0aa4428bf95be148db86b96004634ac4528ef1227ab0cc42"}
Oct 30 03:13:20.977827 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:20.977746    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:13:21.663047 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:21.663013    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerStarted","Data":"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"}
Oct 30 03:13:22.643604 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:22.643578    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:13:24.678792 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:24.678758    4334 generic.go:356] "Generic (PLEG): container finished" podID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerID="6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7" exitCode=0
Oct 30 03:13:24.679212 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:24.678830    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerDied","Data":"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"}
Oct 30 03:13:27.697058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:27.697027    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerStarted","Data":"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"}
Oct 30 03:13:28.705164 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:28.705124    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerStarted","Data":"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"}
Oct 30 03:13:30.718272 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:30.718237    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerStarted","Data":"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"}
Oct 30 03:13:30.787297 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:30.787249    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=14.412288103 podStartE2EDuration="22.78723709s" totalImagesPullingTime="4.787471273s" totalInitContainerRuntime="3.587477714s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:13:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:30.779194473 +0000 UTC m=+1889.785784726" watchObservedRunningTime="2026-10-30 03:13:30.78723709 +0000 UTC m=+1889.793827343"
Oct 30 03:13:35.065734 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:35.065702    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:35.066126 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:35.065774    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:35.070954 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:35.070935    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:35.746237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:35.746212    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:13:44.573986 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:44.573948    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:44.574440 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:44.574224    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" containerName="otc-container" containerID="cri-o://b3bc8f4fd0f7d706bfc451709cc033d02e4869fcdc9ab935a764e895dc292e36" gracePeriod=30
Oct 30 03:13:45.792536 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.792502    4334 generic.go:356] "Generic (PLEG): container finished" podID="b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" containerID="b3bc8f4fd0f7d706bfc451709cc033d02e4869fcdc9ab935a764e895dc292e36" exitCode=0
Oct 30 03:13:45.792886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.792571    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04","Type":"ContainerDied","Data":"b3bc8f4fd0f7d706bfc451709cc033d02e4869fcdc9ab935a764e895dc292e36"}
Oct 30 03:13:45.814669 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.814645    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:45.977907 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.977884    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-kube-api-access-nlqgr\" (UniqueName: \"kubernetes.io/projected/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-kube-api-access-nlqgr\") pod \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") "
Oct 30 03:13:45.978049 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.977971    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-monitoring-tls-certs\") pod \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") "
Oct 30 03:13:45.978049 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.978010    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-tls-certs\") pod \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") "
Oct 30 03:13:45.978049 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.978039    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otc-internal\") pod \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") "
Oct 30 03:13:45.978176 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.978064    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert\") pod \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") "
Oct 30 03:13:45.978176 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.978098    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otlp-tls-certs\") pod \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\" (UID: \"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04\") "
Oct 30 03:13:45.978503 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.978470    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otc-internal" pod "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" (UID: "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:13:45.980291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.980164    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otlp-tls-certs" pod "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" (UID: "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:45.980291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.980209    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-kube-api-access-nlqgr" pod "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" (UID: "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04"). InnerVolumeSpecName "kube-api-access-nlqgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:45.980291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.980265    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-tls-certs" pod "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" (UID: "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:45.980662 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.980628    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert" pod "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" (UID: "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:45.980875 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:45.980855    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-monitoring-tls-certs" pod "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" (UID: "b3513b6a-fcbf-47bd-b6df-fce58e0c4e04"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:13:46.078865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.078837    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.078865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.078860    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.078865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.078870    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.079058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.078878    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.079058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.078889    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.079058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.078897    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqgr\" (UniqueName: \"kubernetes.io/projected/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04-kube-api-access-nlqgr\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:13:46.798525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.798484    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b3513b6a-fcbf-47bd-b6df-fce58e0c4e04","Type":"ContainerDied","Data":"91cc7be09cae6a70c97b6394eff3b9be22cc038b40d8d0b7e5e374ee52e04dc7"}
Oct 30 03:13:46.798979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.798555    4334 scope.go:122] "RemoveContainer" containerID="b3bc8f4fd0f7d706bfc451709cc033d02e4869fcdc9ab935a764e895dc292e36"
Oct 30 03:13:46.798979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.798494    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:46.849120 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.849091    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:46.860779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.860759    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:46.890454 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.890431    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:46.890820 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.890806    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" containerName="otc-container"
Oct 30 03:13:46.890820 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.890821    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" containerName="otc-container"
Oct 30 03:13:46.890941 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.890910    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" containerName="otc-container"
Oct 30 03:13:46.895147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.895132    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:46.912032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.912006    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:13:46.912177 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.912102    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:13:46.912177 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.912041    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:13:46.912177 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.912093    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:13:46.913749 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.913725    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:46.922145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.922126    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:13:46.922245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.922172    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:13:46.988348 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.988322    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwtx\" (UniqueName: \"kubernetes.io/projected/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-kube-api-access-qgwtx\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:46.988437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.988362    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:46.988437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.988386    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:46.988567 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.988546    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:46.988606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.988580    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:46.988686 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:46.988671    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.089106 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.089030    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.089106 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.089073    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.089283 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.089143    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.089283 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.089192    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwtx\" (UniqueName: \"kubernetes.io/projected/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-kube-api-access-qgwtx\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.089283 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.089235    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.089283 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.089267    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.089868 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.089838    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.091693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.091672    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.091802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.091723    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.091865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.091831    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.091981 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.091965    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.107554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.107534    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwtx\" (UniqueName: \"kubernetes.io/projected/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-kube-api-access-qgwtx\") pod \"data-science-collector-collector-1\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.158935 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.158905    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3513b6a-fcbf-47bd-b6df-fce58e0c4e04" path="/var/lib/kubelet/pods/b3513b6a-fcbf-47bd-b6df-fce58e0c4e04/volumes"
Oct 30 03:13:47.207083 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.207054    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:13:47.506104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.506070    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:13:47.509982 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:13:47.509956    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eabe5e7_1010_4811_bac2_9bd7a915c4a4.slice/crio-d91b27462390226147f1d5148c52058723a2ab6e3264042ef5dfac4e2fd826a3 WatchSource:0}: Error finding container d91b27462390226147f1d5148c52058723a2ab6e3264042ef5dfac4e2fd826a3: Status 404 returned error can't find the container with id d91b27462390226147f1d5148c52058723a2ab6e3264042ef5dfac4e2fd826a3
Oct 30 03:13:47.512195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.512163    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:13:47.512290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.512223    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:13:47.806474 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.806384    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0eabe5e7-1010-4811-bac2-9bd7a915c4a4","Type":"ContainerStarted","Data":"534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5"}
Oct 30 03:13:47.806474 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.806436    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0eabe5e7-1010-4811-bac2-9bd7a915c4a4","Type":"ContainerStarted","Data":"d91b27462390226147f1d5148c52058723a2ab6e3264042ef5dfac4e2fd826a3"}
Oct 30 03:13:47.846468 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:13:47.846406    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.846391917 podStartE2EDuration="1.846391917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:47.845255794 +0000 UTC m=+1906.851846062" watchObservedRunningTime="2026-10-30 03:13:47.846391917 +0000 UTC m=+1906.852982172"
Oct 30 03:13:50.978512 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:13:50.978471    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:14:20.978160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:14:20.978087    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:14:50.978596 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:14:50.978555    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:15:10.806629 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:10.806593    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:10.807210 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:10.806845    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0eabe5e7-1010-4811-bac2-9bd7a915c4a4" containerName="otc-container" containerID="cri-o://534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5" gracePeriod=30
Oct 30 03:15:12.089588 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.089556    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:12.092518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.092498    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.107301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.107274    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:15:12.107416 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.107274    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:15:12.107416 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.107344    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:15:12.107558 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.107517    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:15:12.118584 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.118556    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:15:12.118697 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.118612    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-xfppn\""
Oct 30 03:15:12.118697 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.118651    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:15:12.118795 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.118734    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:15:12.120627 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.120608    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:12.256738 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.256711    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.256904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.256753    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.256904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.256827    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.256904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.256874    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz56n\" (UniqueName: \"kubernetes.io/projected/0a9d84d5-f42d-4703-97bd-060549220aa9-kube-api-access-hz56n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.256904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.256902    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.257039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.257013    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.257073 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.257039    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.257073 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.257066    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.358231 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.358146    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.358231 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.358193    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.358231 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.358220    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.358525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.358262    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz56n\" (UniqueName: \"kubernetes.io/projected/0a9d84d5-f42d-4703-97bd-060549220aa9-kube-api-access-hz56n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.358525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.358287    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.358525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.358331    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.358525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.358363    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.358525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.358401    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.359180 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.359157    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.359180 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.359166    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.359332 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.359223    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.359332 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.359298    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.360974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.360949    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.361333 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.361313    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.361399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.361380    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:12.376458 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:12.376415    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz56n\" (UniqueName: \"kubernetes.io/projected/0a9d84d5-f42d-4703-97bd-060549220aa9-kube-api-access-hz56n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:13.145256 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.145233    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.209638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.209554    4334 generic.go:356] "Generic (PLEG): container finished" podID="0eabe5e7-1010-4811-bac2-9bd7a915c4a4" containerID="534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5" exitCode=0
Oct 30 03:15:13.209638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.209599    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.209638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.209617    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0eabe5e7-1010-4811-bac2-9bd7a915c4a4","Type":"ContainerDied","Data":"534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5"}
Oct 30 03:15:13.209875 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.209660    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0eabe5e7-1010-4811-bac2-9bd7a915c4a4","Type":"ContainerDied","Data":"d91b27462390226147f1d5148c52058723a2ab6e3264042ef5dfac4e2fd826a3"}
Oct 30 03:15:13.209875 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.209683    4334 scope.go:122] "RemoveContainer" containerID="534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5"
Oct 30 03:15:13.218055 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.218038    4334 scope.go:122] "RemoveContainer" containerID="534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5"
Oct 30 03:15:13.218345 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:15:13.218325    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5\": container with ID starting with 534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5 not found: ID does not exist" containerID="534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5"
Oct 30 03:15:13.218399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.218355    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5"} err="failed to get container status \"534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5\": rpc error: code = NotFound desc = could not find container \"534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5\": container with ID starting with 534b08a79164322e1f96ae0dafcb8fa18ac89144ae0e7c713b0d96bbd7e9f8e5 not found: ID does not exist"
Oct 30 03:15:13.267005 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.266984    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-kube-api-access-qgwtx\" (UniqueName: \"kubernetes.io/projected/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-kube-api-access-qgwtx\") pod \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") "
Oct 30 03:15:13.267125 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.267030    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-tls-certs\") pod \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") "
Oct 30 03:15:13.267125 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.267098    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-monitoring-tls-certs\") pod \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") "
Oct 30 03:15:13.267287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.267135    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-data-science-collector-ta-client-cert\") pod \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") "
Oct 30 03:15:13.267287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.267186    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otlp-tls-certs\") pod \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") "
Oct 30 03:15:13.267287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.267223    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otc-internal\") pod \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\" (UID: \"0eabe5e7-1010-4811-bac2-9bd7a915c4a4\") "
Oct 30 03:15:13.267871 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.267818    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otc-internal" pod "0eabe5e7-1010-4811-bac2-9bd7a915c4a4" (UID: "0eabe5e7-1010-4811-bac2-9bd7a915c4a4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:13.269638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.269610    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-monitoring-tls-certs" pod "0eabe5e7-1010-4811-bac2-9bd7a915c4a4" (UID: "0eabe5e7-1010-4811-bac2-9bd7a915c4a4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:13.269638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.269622    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-data-science-collector-ta-client-cert" pod "0eabe5e7-1010-4811-bac2-9bd7a915c4a4" (UID: "0eabe5e7-1010-4811-bac2-9bd7a915c4a4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:13.269780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.269759    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otlp-tls-certs" pod "0eabe5e7-1010-4811-bac2-9bd7a915c4a4" (UID: "0eabe5e7-1010-4811-bac2-9bd7a915c4a4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:13.269820 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.269784    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-tls-certs" pod "0eabe5e7-1010-4811-bac2-9bd7a915c4a4" (UID: "0eabe5e7-1010-4811-bac2-9bd7a915c4a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:13.269906 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.269891    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-kube-api-access-qgwtx" pod "0eabe5e7-1010-4811-bac2-9bd7a915c4a4" (UID: "0eabe5e7-1010-4811-bac2-9bd7a915c4a4"). InnerVolumeSpecName "kube-api-access-qgwtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:13.368095 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.368072    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:13.368198 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.368096    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:13.368198 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.368109    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:13.368198 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.368117    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:13.368198 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.368126    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwtx\" (UniqueName: \"kubernetes.io/projected/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-kube-api-access-qgwtx\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:13.368198 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.368137    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eabe5e7-1010-4811-bac2-9bd7a915c4a4-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:13.551250 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.551220    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:13.562286 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.562255    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:13.591603 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.591578    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:13.592252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.592232    4334 cpu_manager.go:414] "containerMap: removing container" podUID="0eabe5e7-1010-4811-bac2-9bd7a915c4a4" containerName="otc-container"
Oct 30 03:15:13.592252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.592251    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eabe5e7-1010-4811-bac2-9bd7a915c4a4" containerName="otc-container"
Oct 30 03:15:13.592407 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.592310    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eabe5e7-1010-4811-bac2-9bd7a915c4a4" containerName="otc-container"
Oct 30 03:15:13.595119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.595103    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.610714 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.610662    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\""
Oct 30 03:15:13.610849 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.610772    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:13.610966 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.610850    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:13.610966 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.610925    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:13.613117 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.613096    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:13.621291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.621271    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:13.621291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.621280    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:13.671120 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.671092    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.671236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.671133    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.671236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.671156    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.671314 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.671276    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.671349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.671338    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.671419 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.671401    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnvf\" (UniqueName: \"kubernetes.io/projected/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-kube-api-access-blnvf\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.772810 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.772730    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.772810 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.772776    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.772810 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.772808    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnvf\" (UniqueName: \"kubernetes.io/projected/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-kube-api-access-blnvf\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.773085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.772857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.773085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.772882    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.773085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.772904    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.773721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.773690    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.775457 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.775421    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.775552 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.775494    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.775605 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.775591    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.775658 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.775636    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.790638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.790619    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnvf\" (UniqueName: \"kubernetes.io/projected/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-kube-api-access-blnvf\") pod \"data-science-collector-collector-1\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:13.907894 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:13.907864    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:14.200564 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:14.200534    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:14.204767 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:15:14.204735    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1baf0ad_af81_4d1d_8300_5d90b7c9cbbc.slice/crio-ae1ef23016add03b20a0b24e43f3bc6c5f3d31f15e11a3918116f085a33c455e WatchSource:0}: Error finding container ae1ef23016add03b20a0b24e43f3bc6c5f3d31f15e11a3918116f085a33c455e: Status 404 returned error can't find the container with id ae1ef23016add03b20a0b24e43f3bc6c5f3d31f15e11a3918116f085a33c455e
Oct 30 03:15:14.206939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:14.206901    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:14.207027 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:14.206964    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:14.217970 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:14.217936    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc","Type":"ContainerStarted","Data":"ae1ef23016add03b20a0b24e43f3bc6c5f3d31f15e11a3918116f085a33c455e"}
Oct 30 03:15:15.159266 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:15.159231    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eabe5e7-1010-4811-bac2-9bd7a915c4a4" path="/var/lib/kubelet/pods/0eabe5e7-1010-4811-bac2-9bd7a915c4a4/volumes"
Oct 30 03:15:15.223754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:15.223722    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc","Type":"ContainerStarted","Data":"c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739"}
Oct 30 03:15:15.262437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:15.262393    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.262380643 podStartE2EDuration="2.262380643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:15.260432921 +0000 UTC m=+1994.267023174" watchObservedRunningTime="2026-10-30 03:15:15.262380643 +0000 UTC m=+1994.268970897"
Oct 30 03:15:20.249223 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.249191    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:20.249848 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.249414    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" containerName="otc-container" containerID="cri-o://c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739" gracePeriod=30
Oct 30 03:15:20.487640 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.487610    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:20.636338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.636247    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otc-internal\") pod \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") "
Oct 30 03:15:20.636338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.636326    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-monitoring-tls-certs\") pod \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") "
Oct 30 03:15:20.636589 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.636353    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-data-science-collector-ta-client-cert\") pod \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") "
Oct 30 03:15:20.636589 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.636390    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-kube-api-access-blnvf\" (UniqueName: \"kubernetes.io/projected/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-kube-api-access-blnvf\") pod \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") "
Oct 30 03:15:20.636589 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.636426    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otlp-tls-certs\") pod \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") "
Oct 30 03:15:20.636589 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.636499    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-tls-certs\") pod \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\" (UID: \"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc\") "
Oct 30 03:15:20.636794 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.636708    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otc-internal" pod "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" (UID: "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.636853 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.636841    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.638795 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.638763    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-data-science-collector-ta-client-cert" pod "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" (UID: "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:20.639049 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.639027    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-monitoring-tls-certs" pod "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" (UID: "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:20.639156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.639131    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-kube-api-access-blnvf" pod "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" (UID: "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc"). InnerVolumeSpecName "kube-api-access-blnvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.639233 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.639208    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otlp-tls-certs" pod "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" (UID: "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:20.639233 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.639224    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-tls-certs" pod "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" (UID: "c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:20.737426 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.737397    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.737426 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.737422    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.737636 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.737434    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blnvf\" (UniqueName: \"kubernetes.io/projected/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-kube-api-access-blnvf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.737636 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.737464    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.737636 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:20.737474    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.978223 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:15:20.978186    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:15:21.251162 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.251085    4334 generic.go:356] "Generic (PLEG): container finished" podID="c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" containerID="c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739" exitCode=0
Oct 30 03:15:21.251162 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.251131    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc","Type":"ContainerDied","Data":"c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739"}
Oct 30 03:15:21.251699 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.251165    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.251699 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.251184    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc","Type":"ContainerDied","Data":"ae1ef23016add03b20a0b24e43f3bc6c5f3d31f15e11a3918116f085a33c455e"}
Oct 30 03:15:21.251699 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.251208    4334 scope.go:122] "RemoveContainer" containerID="c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739"
Oct 30 03:15:21.259821 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.259803    4334 scope.go:122] "RemoveContainer" containerID="c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739"
Oct 30 03:15:21.260089 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:15:21.260069    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739\": container with ID starting with c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739 not found: ID does not exist" containerID="c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739"
Oct 30 03:15:21.260152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.260103    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739"} err="failed to get container status \"c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739\": rpc error: code = NotFound desc = could not find container \"c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739\": container with ID starting with c9e72d05f4798f1bfe3f2c8a52c3328b94b907e4a67329f5c620ba58163c5739 not found: ID does not exist"
Oct 30 03:15:21.301804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.301772    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:21.313141 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.313117    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:21.349194 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.349171    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:21.349548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.349530    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" containerName="otc-container"
Oct 30 03:15:21.349548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.349546    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" containerName="otc-container"
Oct 30 03:15:21.349685 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.349625    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" containerName="otc-container"
Oct 30 03:15:21.352591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.352574    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.372087 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.372061    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:15:21.372414 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.372392    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:21.372615 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.372593    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:21.372615 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.372602    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:21.372764 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.372608    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:21.372764 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.372696    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:21.380114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.380099    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:21.443730 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.443702    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f58f23f1-f393-4d4b-bc33-487e28e98746-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.443828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.443743    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82f2\" (UniqueName: \"kubernetes.io/projected/f58f23f1-f393-4d4b-bc33-487e28e98746-kube-api-access-l82f2\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.443828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.443777    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.443828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.443815    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.443932 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.443833    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.443932 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.443857    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.486800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.486768    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:21.500080 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:15:21.500052    4334 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[tempo-storage], unattached volumes=[tempo-storage], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="0a9d84d5-f42d-4703-97bd-060549220aa9"
Oct 30 03:15:21.544430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.544356    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f58f23f1-f393-4d4b-bc33-487e28e98746-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.544430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.544401    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82f2\" (UniqueName: \"kubernetes.io/projected/f58f23f1-f393-4d4b-bc33-487e28e98746-kube-api-access-l82f2\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.544430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.544425    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.544719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.544443    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.544719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.544489    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.544719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.544580    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.545156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.545128    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f58f23f1-f393-4d4b-bc33-487e28e98746-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.547022 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.546992    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.547117 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.547029    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.547399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.547378    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.547464 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.547431    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.562995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.562972    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82f2\" (UniqueName: \"kubernetes.io/projected/f58f23f1-f393-4d4b-bc33-487e28e98746-kube-api-access-l82f2\") pod \"data-science-collector-collector-1\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.665642 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.665618    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:21.960569 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.960531    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:21.963677 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:15:21.963647    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58f23f1_f393_4d4b_bc33_487e28e98746.slice/crio-b84cc232afa52791f1a57abcedbf0673daeaa70bc99aaac9e9a86cb7399cf9eb WatchSource:0}: Error finding container b84cc232afa52791f1a57abcedbf0673daeaa70bc99aaac9e9a86cb7399cf9eb: Status 404 returned error can't find the container with id b84cc232afa52791f1a57abcedbf0673daeaa70bc99aaac9e9a86cb7399cf9eb
Oct 30 03:15:21.965993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.965963    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:21.966062 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:21.966023    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:22.259292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.259219    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:22.259292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.259237    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f58f23f1-f393-4d4b-bc33-487e28e98746","Type":"ContainerStarted","Data":"7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f"}
Oct 30 03:15:22.259292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.259287    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f58f23f1-f393-4d4b-bc33-487e28e98746","Type":"ContainerStarted","Data":"b84cc232afa52791f1a57abcedbf0673daeaa70bc99aaac9e9a86cb7399cf9eb"}
Oct 30 03:15:22.264608 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.264588    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:22.343902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.343852    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.343839668 podStartE2EDuration="1.343839668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:22.342258121 +0000 UTC m=+2001.348848353" watchObservedRunningTime="2026-10-30 03:15:22.343839668 +0000 UTC m=+2001.350429922"
Oct 30 03:15:22.452153 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452119    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9d84d5-f42d-4703-97bd-060549220aa9-kube-api-access-hz56n\" (UniqueName: \"kubernetes.io/projected/0a9d84d5-f42d-4703-97bd-060549220aa9-kube-api-access-hz56n\") pod \"0a9d84d5-f42d-4703-97bd-060549220aa9\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") "
Oct 30 03:15:22.452329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452169    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-rbac\") pod \"0a9d84d5-f42d-4703-97bd-060549220aa9\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") "
Oct 30 03:15:22.452329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452194    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"0a9d84d5-f42d-4703-97bd-060549220aa9\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") "
Oct 30 03:15:22.452329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452238    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"0a9d84d5-f42d-4703-97bd-060549220aa9\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") "
Oct 30 03:15:22.452537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452386    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-conf\") pod \"0a9d84d5-f42d-4703-97bd-060549220aa9\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") "
Oct 30 03:15:22.452537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452478    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-mtls\") pod \"0a9d84d5-f42d-4703-97bd-060549220aa9\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") "
Oct 30 03:15:22.452537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452520    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"0a9d84d5-f42d-4703-97bd-060549220aa9\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") "
Oct 30 03:15:22.452711 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452571    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-tenants\") pod \"0a9d84d5-f42d-4703-97bd-060549220aa9\" (UID: \"0a9d84d5-f42d-4703-97bd-060549220aa9\") "
Oct 30 03:15:22.452711 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452587    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-rbac" pod "0a9d84d5-f42d-4703-97bd-060549220aa9" (UID: "0a9d84d5-f42d-4703-97bd-060549220aa9"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.452803 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452674    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-ca-bundle" pod "0a9d84d5-f42d-4703-97bd-060549220aa9" (UID: "0a9d84d5-f42d-4703-97bd-060549220aa9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.453017 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452955    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.453017 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452980    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.453017 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.452986    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-conf" pod "0a9d84d5-f42d-4703-97bd-060549220aa9" (UID: "0a9d84d5-f42d-4703-97bd-060549220aa9"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.453200 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.453082    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-serving-cabundle" pod "0a9d84d5-f42d-4703-97bd-060549220aa9" (UID: "0a9d84d5-f42d-4703-97bd-060549220aa9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:22.454689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.454662    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-mtls" pod "0a9d84d5-f42d-4703-97bd-060549220aa9" (UID: "0a9d84d5-f42d-4703-97bd-060549220aa9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:22.454689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.454678    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "0a9d84d5-f42d-4703-97bd-060549220aa9" (UID: "0a9d84d5-f42d-4703-97bd-060549220aa9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:22.454834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.454779    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9d84d5-f42d-4703-97bd-060549220aa9-kube-api-access-hz56n" pod "0a9d84d5-f42d-4703-97bd-060549220aa9" (UID: "0a9d84d5-f42d-4703-97bd-060549220aa9"). InnerVolumeSpecName "kube-api-access-hz56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:22.454917 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.454898    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-tenants" pod "0a9d84d5-f42d-4703-97bd-060549220aa9" (UID: "0a9d84d5-f42d-4703-97bd-060549220aa9"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:22.553728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.553387    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.553728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.553422    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.553728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.553439    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.553728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.553473    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-gateway-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.553728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.553488    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz56n\" (UniqueName: \"kubernetes.io/projected/0a9d84d5-f42d-4703-97bd-060549220aa9-kube-api-access-hz56n\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:22.553728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:22.553503    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a9d84d5-f42d-4703-97bd-060549220aa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:23.159063 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:23.159030    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc" path="/var/lib/kubelet/pods/c1baf0ad-af81-4d1d-8300-5d90b7c9cbbc/volumes"
Oct 30 03:15:23.264176 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:23.264149    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:23.377843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:23.377804    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:23.391398 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:23.391371    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:25.158632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:25.158598    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9d84d5-f42d-4703-97bd-060549220aa9" path="/var/lib/kubelet/pods/0a9d84d5-f42d-4703-97bd-060549220aa9/volumes"
Oct 30 03:15:29.316143 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.316108    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:29.316620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.316327    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f58f23f1-f393-4d4b-bc33-487e28e98746" containerName="otc-container" containerID="cri-o://7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f" gracePeriod=30
Oct 30 03:15:29.560623 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.560600    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:29.614085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.613831    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-tls-certs\") pod \"f58f23f1-f393-4d4b-bc33-487e28e98746\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") "
Oct 30 03:15:29.614085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.613896    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58f23f1-f393-4d4b-bc33-487e28e98746-kube-api-access-l82f2\" (UniqueName: \"kubernetes.io/projected/f58f23f1-f393-4d4b-bc33-487e28e98746-kube-api-access-l82f2\") pod \"f58f23f1-f393-4d4b-bc33-487e28e98746\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") "
Oct 30 03:15:29.614085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.613924    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-monitoring-tls-certs\") pod \"f58f23f1-f393-4d4b-bc33-487e28e98746\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") "
Oct 30 03:15:29.614085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.613954    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-data-science-collector-ta-client-cert\") pod \"f58f23f1-f393-4d4b-bc33-487e28e98746\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") "
Oct 30 03:15:29.614085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.614017    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-otlp-tls-certs\") pod \"f58f23f1-f393-4d4b-bc33-487e28e98746\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") "
Oct 30 03:15:29.614412 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.614090    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58f23f1-f393-4d4b-bc33-487e28e98746-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f58f23f1-f393-4d4b-bc33-487e28e98746-otc-internal\") pod \"f58f23f1-f393-4d4b-bc33-487e28e98746\" (UID: \"f58f23f1-f393-4d4b-bc33-487e28e98746\") "
Oct 30 03:15:29.614919 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.614839    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58f23f1-f393-4d4b-bc33-487e28e98746-otc-internal" pod "f58f23f1-f393-4d4b-bc33-487e28e98746" (UID: "f58f23f1-f393-4d4b-bc33-487e28e98746"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:29.617035 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.617004    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-monitoring-tls-certs" pod "f58f23f1-f393-4d4b-bc33-487e28e98746" (UID: "f58f23f1-f393-4d4b-bc33-487e28e98746"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:29.617035 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.617026    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-data-science-collector-ta-client-cert" pod "f58f23f1-f393-4d4b-bc33-487e28e98746" (UID: "f58f23f1-f393-4d4b-bc33-487e28e98746"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:29.617253 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.617008    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58f23f1-f393-4d4b-bc33-487e28e98746-kube-api-access-l82f2" pod "f58f23f1-f393-4d4b-bc33-487e28e98746" (UID: "f58f23f1-f393-4d4b-bc33-487e28e98746"). InnerVolumeSpecName "kube-api-access-l82f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:29.617425 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.617397    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-otlp-tls-certs" pod "f58f23f1-f393-4d4b-bc33-487e28e98746" (UID: "f58f23f1-f393-4d4b-bc33-487e28e98746"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:29.618805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.618785    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-tls-certs" pod "f58f23f1-f393-4d4b-bc33-487e28e98746" (UID: "f58f23f1-f393-4d4b-bc33-487e28e98746"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:29.715097 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.715070    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:29.715097 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.715092    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f58f23f1-f393-4d4b-bc33-487e28e98746-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:29.715097 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.715101    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:29.715288 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.715112    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l82f2\" (UniqueName: \"kubernetes.io/projected/f58f23f1-f393-4d4b-bc33-487e28e98746-kube-api-access-l82f2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:29.715288 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.715121    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:29.715288 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:29.715130    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f58f23f1-f393-4d4b-bc33-487e28e98746-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:30.298037 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.298003    4334 generic.go:356] "Generic (PLEG): container finished" podID="f58f23f1-f393-4d4b-bc33-487e28e98746" containerID="7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f" exitCode=0
Oct 30 03:15:30.298211 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.298063    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f58f23f1-f393-4d4b-bc33-487e28e98746","Type":"ContainerDied","Data":"7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f"}
Oct 30 03:15:30.298211 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.298075    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.298211 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.298104    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f58f23f1-f393-4d4b-bc33-487e28e98746","Type":"ContainerDied","Data":"b84cc232afa52791f1a57abcedbf0673daeaa70bc99aaac9e9a86cb7399cf9eb"}
Oct 30 03:15:30.298211 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.298127    4334 scope.go:122] "RemoveContainer" containerID="7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f"
Oct 30 03:15:30.311532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.311504    4334 scope.go:122] "RemoveContainer" containerID="7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f"
Oct 30 03:15:30.311997 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:15:30.311971    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f\": container with ID starting with 7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f not found: ID does not exist" containerID="7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f"
Oct 30 03:15:30.312120 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.312007    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f"} err="failed to get container status \"7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f\": rpc error: code = NotFound desc = could not find container \"7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f\": container with ID starting with 7940faf4b940a1d3dfd9c25c2627335e42a0b89c7e9d33f948b451ae5262e73f not found: ID does not exist"
Oct 30 03:15:30.345949 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.345911    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:30.356995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.356973    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:30.385786 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.385721    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:30.386137 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.386122    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f58f23f1-f393-4d4b-bc33-487e28e98746" containerName="otc-container"
Oct 30 03:15:30.386195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.386139    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58f23f1-f393-4d4b-bc33-487e28e98746" containerName="otc-container"
Oct 30 03:15:30.386250 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.386209    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58f23f1-f393-4d4b-bc33-487e28e98746" containerName="otc-container"
Oct 30 03:15:30.389179 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.389165    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.408252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.405979    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\""
Oct 30 03:15:30.408252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.406473    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:30.408252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.406480    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:30.408252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.406490    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:30.408252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.406589    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:30.408252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.407702    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:30.415725 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.415706    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:30.420126 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.420106    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.420213 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.420143    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.420213 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.420161    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrklc\" (UniqueName: \"kubernetes.io/projected/2127890e-19db-4608-9e64-36573a71ee02-kube-api-access-wrklc\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.420283 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.420214    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.420332 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.420321    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.420368 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.420346    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2127890e-19db-4608-9e64-36573a71ee02-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.521038 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.520987    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.521038 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.521023    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrklc\" (UniqueName: \"kubernetes.io/projected/2127890e-19db-4608-9e64-36573a71ee02-kube-api-access-wrklc\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.521290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.521049    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.521290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.521090    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.521290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.521115    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2127890e-19db-4608-9e64-36573a71ee02-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.521290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.521163    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.522077 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.522046    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2127890e-19db-4608-9e64-36573a71ee02-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.523608 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.523580    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.523700 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.523675    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.523762 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.523739    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.523845 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.523831    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.538667 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.538645    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrklc\" (UniqueName: \"kubernetes.io/projected/2127890e-19db-4608-9e64-36573a71ee02-kube-api-access-wrklc\") pod \"data-science-collector-collector-1\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.700711 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.700685    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:30.732402 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.732370    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:30.735693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.735669    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:30.749886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.749854    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:15:30.749886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.749870    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:15:30.749886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.749856    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:15:30.750110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.749908    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:15:30.750110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.749915    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:15:30.750110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.750012    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:15:30.750110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.750056    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-qptjc\""
Oct 30 03:15:30.750110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.750084    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:15:30.761019 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.760905    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:30.926655 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.926621    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:30.926818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.926659    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:30.926818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.926773    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:30.926818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.926800    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:30.926949 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.926891    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlt9q\" (UniqueName: \"kubernetes.io/projected/f27060c3-32c9-4132-b8d2-2aa776a69d12-kube-api-access-nlt9q\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:30.926949 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.926944    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:30.927036 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.926985    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:30.927096 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:30.927078    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.005620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.005547    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:31.009566 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:15:31.009538    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2127890e_19db_4608_9e64_36573a71ee02.slice/crio-bf9acee61b200f04f3ce36ddd06f2bf4580e89d86d6b1a995ebebdba29156be0 WatchSource:0}: Error finding container bf9acee61b200f04f3ce36ddd06f2bf4580e89d86d6b1a995ebebdba29156be0: Status 404 returned error can't find the container with id bf9acee61b200f04f3ce36ddd06f2bf4580e89d86d6b1a995ebebdba29156be0
Oct 30 03:15:31.011818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.011785    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:31.011917 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.011852    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:31.027484 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.027438    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.027565 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.027507    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.027565 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.027531    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlt9q\" (UniqueName: \"kubernetes.io/projected/f27060c3-32c9-4132-b8d2-2aa776a69d12-kube-api-access-nlt9q\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.027565 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.027558    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.027715 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.027597    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.027715 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.027625    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.027715 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.027689    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.027875 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.027719    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.028125 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.028092    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.028240 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.028217    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.028336 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.028315    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.028481 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.028459    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.030023 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.030002    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.030151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.030131    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.030190 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.030166    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.045282 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.045257    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlt9q\" (UniqueName: \"kubernetes.io/projected/f27060c3-32c9-4132-b8d2-2aa776a69d12-kube-api-access-nlt9q\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:31.159849 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.159819    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58f23f1-f393-4d4b-bc33-487e28e98746" path="/var/lib/kubelet/pods/f58f23f1-f393-4d4b-bc33-487e28e98746/volumes"
Oct 30 03:15:31.306791 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.306715    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2127890e-19db-4608-9e64-36573a71ee02","Type":"ContainerStarted","Data":"0ea1b7068a2f6143e35ede08f9b8381b969c7eccb928e1ea1b65a03e05c5a2ca"}
Oct 30 03:15:31.306791 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.306765    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2127890e-19db-4608-9e64-36573a71ee02","Type":"ContainerStarted","Data":"bf9acee61b200f04f3ce36ddd06f2bf4580e89d86d6b1a995ebebdba29156be0"}
Oct 30 03:15:31.346171 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:31.346126    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.346111479 podStartE2EDuration="1.346111479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:31.345786711 +0000 UTC m=+2010.352376966" watchObservedRunningTime="2026-10-30 03:15:31.346111479 +0000 UTC m=+2010.352701733"
Oct 30 03:15:33.146141 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.146107    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:33.175730 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.175703    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:33.246738 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.246711    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:33.246864 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.246820    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:33.259904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.259881    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:33.361375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.361341    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:33.389055 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.389030    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:33.452571 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.452548    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:33.775004 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:33.774917    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:33.778611 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:15:33.778574    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27060c3_32c9_4132_b8d2_2aa776a69d12.slice/crio-827800f665b1d549b530a65f5d8de57359c3d0b9000229f30ebaf85063e7b5a6 WatchSource:0}: Error finding container 827800f665b1d549b530a65f5d8de57359c3d0b9000229f30ebaf85063e7b5a6: Status 404 returned error can't find the container with id 827800f665b1d549b530a65f5d8de57359c3d0b9000229f30ebaf85063e7b5a6
Oct 30 03:15:34.323871 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:34.323838    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f27060c3-32c9-4132-b8d2-2aa776a69d12","Type":"ContainerStarted","Data":"0084a33835c4b5a0b80cc6ec9046fb5edfa45ba7c0656b9cd831b8bab2de5f8c"}
Oct 30 03:15:34.324369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:34.323886    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f27060c3-32c9-4132-b8d2-2aa776a69d12","Type":"ContainerStarted","Data":"a5003a535af68c251ac57737068f7521f2c62e4a2aaa809cd480a725784d8b07"}
Oct 30 03:15:34.324369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:34.323904    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f27060c3-32c9-4132-b8d2-2aa776a69d12","Type":"ContainerStarted","Data":"336babf49a9ebecae200e1a296e14f3ab6b9fc6cae93a7cd8d57fb0d82f4687a"}
Oct 30 03:15:34.324369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:34.323918    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f27060c3-32c9-4132-b8d2-2aa776a69d12","Type":"ContainerStarted","Data":"827800f665b1d549b530a65f5d8de57359c3d0b9000229f30ebaf85063e7b5a6"}
Oct 30 03:15:34.324369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:34.324067    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:34.326749 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:34.326721    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:34.388276 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:34.388219    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.388203427 podStartE2EDuration="4.388203427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:34.375223566 +0000 UTC m=+2013.381813819" watchObservedRunningTime="2026-10-30 03:15:34.388203427 +0000 UTC m=+2013.394793684"
Oct 30 03:15:35.328357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:35.328321    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:35.328780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:35.328388    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:35.335838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:35.335816    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:38.165871 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.165831    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:38.166761 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.166117    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2127890e-19db-4608-9e64-36573a71ee02" containerName="otc-container" containerID="cri-o://0ea1b7068a2f6143e35ede08f9b8381b969c7eccb928e1ea1b65a03e05c5a2ca" gracePeriod=30
Oct 30 03:15:38.345292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.345264    4334 generic.go:356] "Generic (PLEG): container finished" podID="2127890e-19db-4608-9e64-36573a71ee02" containerID="0ea1b7068a2f6143e35ede08f9b8381b969c7eccb928e1ea1b65a03e05c5a2ca" exitCode=0
Oct 30 03:15:38.345427 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.345343    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2127890e-19db-4608-9e64-36573a71ee02","Type":"ContainerDied","Data":"0ea1b7068a2f6143e35ede08f9b8381b969c7eccb928e1ea1b65a03e05c5a2ca"}
Oct 30 03:15:38.420724 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.420669    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:38.594336 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.594303    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-data-science-collector-ta-client-cert\") pod \"2127890e-19db-4608-9e64-36573a71ee02\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") "
Oct 30 03:15:38.594554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.594345    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-monitoring-tls-certs\") pod \"2127890e-19db-4608-9e64-36573a71ee02\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") "
Oct 30 03:15:38.594554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.594380    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-otlp-tls-certs\") pod \"2127890e-19db-4608-9e64-36573a71ee02\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") "
Oct 30 03:15:38.594554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.594493    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2127890e-19db-4608-9e64-36573a71ee02-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2127890e-19db-4608-9e64-36573a71ee02-otc-internal\") pod \"2127890e-19db-4608-9e64-36573a71ee02\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") "
Oct 30 03:15:38.594755 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.594602    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2127890e-19db-4608-9e64-36573a71ee02-kube-api-access-wrklc\" (UniqueName: \"kubernetes.io/projected/2127890e-19db-4608-9e64-36573a71ee02-kube-api-access-wrklc\") pod \"2127890e-19db-4608-9e64-36573a71ee02\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") "
Oct 30 03:15:38.594755 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.594634    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-tls-certs\") pod \"2127890e-19db-4608-9e64-36573a71ee02\" (UID: \"2127890e-19db-4608-9e64-36573a71ee02\") "
Oct 30 03:15:38.594915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.594887    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2127890e-19db-4608-9e64-36573a71ee02-otc-internal" pod "2127890e-19db-4608-9e64-36573a71ee02" (UID: "2127890e-19db-4608-9e64-36573a71ee02"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:38.596722 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.596694    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-monitoring-tls-certs" pod "2127890e-19db-4608-9e64-36573a71ee02" (UID: "2127890e-19db-4608-9e64-36573a71ee02"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:38.596832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.596744    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-tls-certs" pod "2127890e-19db-4608-9e64-36573a71ee02" (UID: "2127890e-19db-4608-9e64-36573a71ee02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:38.596832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.596778    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2127890e-19db-4608-9e64-36573a71ee02-kube-api-access-wrklc" pod "2127890e-19db-4608-9e64-36573a71ee02" (UID: "2127890e-19db-4608-9e64-36573a71ee02"). InnerVolumeSpecName "kube-api-access-wrklc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:38.597011 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.596995    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-data-science-collector-ta-client-cert" pod "2127890e-19db-4608-9e64-36573a71ee02" (UID: "2127890e-19db-4608-9e64-36573a71ee02"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:38.597070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.597013    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-otlp-tls-certs" pod "2127890e-19db-4608-9e64-36573a71ee02" (UID: "2127890e-19db-4608-9e64-36573a71ee02"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:38.696242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.696174    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2127890e-19db-4608-9e64-36573a71ee02-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:38.696242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.696200    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrklc\" (UniqueName: \"kubernetes.io/projected/2127890e-19db-4608-9e64-36573a71ee02-kube-api-access-wrklc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:38.696242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.696210    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:38.696242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.696227    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:38.696242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.696236    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:38.696242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:38.696244    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2127890e-19db-4608-9e64-36573a71ee02-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:39.215595 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.215561    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:39.216002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.215814    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo" containerID="cri-o://336babf49a9ebecae200e1a296e14f3ab6b9fc6cae93a7cd8d57fb0d82f4687a" gracePeriod=30
Oct 30 03:15:39.216230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.216203    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway-opa" containerID="cri-o://0084a33835c4b5a0b80cc6ec9046fb5edfa45ba7c0656b9cd831b8bab2de5f8c" gracePeriod=30
Oct 30 03:15:39.216336 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.216319    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway" containerID="cri-o://a5003a535af68c251ac57737068f7521f2c62e4a2aaa809cd480a725784d8b07" gracePeriod=30
Oct 30 03:15:39.325349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.325304    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.44:8083/ready\": dial tcp 10.133.0.44:8083: connect: connection refused"
Oct 30 03:15:39.352831 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.352800    4334 generic.go:356] "Generic (PLEG): container finished" podID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerID="0084a33835c4b5a0b80cc6ec9046fb5edfa45ba7c0656b9cd831b8bab2de5f8c" exitCode=0
Oct 30 03:15:39.352956 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.352871    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f27060c3-32c9-4132-b8d2-2aa776a69d12","Type":"ContainerDied","Data":"0084a33835c4b5a0b80cc6ec9046fb5edfa45ba7c0656b9cd831b8bab2de5f8c"}
Oct 30 03:15:39.354182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.354167    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.354256 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.354195    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2127890e-19db-4608-9e64-36573a71ee02","Type":"ContainerDied","Data":"bf9acee61b200f04f3ce36ddd06f2bf4580e89d86d6b1a995ebebdba29156be0"}
Oct 30 03:15:39.354256 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.354242    4334 scope.go:122] "RemoveContainer" containerID="0ea1b7068a2f6143e35ede08f9b8381b969c7eccb928e1ea1b65a03e05c5a2ca"
Oct 30 03:15:39.409308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.409270    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:39.410984 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.410961    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:39.444366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.444335    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:39.444920 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.444891    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2127890e-19db-4608-9e64-36573a71ee02" containerName="otc-container"
Oct 30 03:15:39.444920 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.444914    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2127890e-19db-4608-9e64-36573a71ee02" containerName="otc-container"
Oct 30 03:15:39.445046 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.444994    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2127890e-19db-4608-9e64-36573a71ee02" containerName="otc-container"
Oct 30 03:15:39.446979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.446961    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.462874 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.462709    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:39.462874 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.462781    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:39.463086 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.463066    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:39.463171 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.463158    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:15:39.463247 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.463233    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:39.463327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.463314    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:39.465132 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.465108    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:39.605070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.605035    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.605212 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.605132    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.605212 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.605168    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.605292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.605270    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/89727543-0048-4117-9882-19efe5a1c80c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.605327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.605293    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgf46\" (UniqueName: \"kubernetes.io/projected/89727543-0048-4117-9882-19efe5a1c80c-kube-api-access-xgf46\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.605327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.605310    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.706035 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.705958    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.706035 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.705998    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.706035 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.706024    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.706301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.706066    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/89727543-0048-4117-9882-19efe5a1c80c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.706301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.706081    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgf46\" (UniqueName: \"kubernetes.io/projected/89727543-0048-4117-9882-19efe5a1c80c-kube-api-access-xgf46\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.706301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.706096    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.706954 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.706924    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/89727543-0048-4117-9882-19efe5a1c80c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.708525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.708497    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.708632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.708551    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.708677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.708636    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.708715 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.708675    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.724082 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.724060    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgf46\" (UniqueName: \"kubernetes.io/projected/89727543-0048-4117-9882-19efe5a1c80c-kube-api-access-xgf46\") pod \"data-science-collector-collector-1\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:39.759380 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:39.759359    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:40.052011 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.051931    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:40.058026 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:15:40.057983    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89727543_0048_4117_9882_19efe5a1c80c.slice/crio-b45eaf2b22d7276a635cd3590a06e2be111b16e4117790ebfaaeb36e5f4708b9 WatchSource:0}: Error finding container b45eaf2b22d7276a635cd3590a06e2be111b16e4117790ebfaaeb36e5f4708b9: Status 404 returned error can't find the container with id b45eaf2b22d7276a635cd3590a06e2be111b16e4117790ebfaaeb36e5f4708b9
Oct 30 03:15:40.060290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.060257    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:40.060351 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.060321    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:40.329334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.329269    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.133.0.44:8081/ready\": dial tcp 10.133.0.44:8081: connect: connection refused"
Oct 30 03:15:40.362030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.362001    4334 generic.go:356] "Generic (PLEG): container finished" podID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerID="a5003a535af68c251ac57737068f7521f2c62e4a2aaa809cd480a725784d8b07" exitCode=0
Oct 30 03:15:40.362030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.362026    4334 generic.go:356] "Generic (PLEG): container finished" podID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerID="336babf49a9ebecae200e1a296e14f3ab6b9fc6cae93a7cd8d57fb0d82f4687a" exitCode=0
Oct 30 03:15:40.362277 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.362086    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f27060c3-32c9-4132-b8d2-2aa776a69d12","Type":"ContainerDied","Data":"a5003a535af68c251ac57737068f7521f2c62e4a2aaa809cd480a725784d8b07"}
Oct 30 03:15:40.362277 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.362151    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f27060c3-32c9-4132-b8d2-2aa776a69d12","Type":"ContainerDied","Data":"336babf49a9ebecae200e1a296e14f3ab6b9fc6cae93a7cd8d57fb0d82f4687a"}
Oct 30 03:15:40.363984 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.363918    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"89727543-0048-4117-9882-19efe5a1c80c","Type":"ContainerStarted","Data":"8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12"}
Oct 30 03:15:40.363984 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.363966    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"89727543-0048-4117-9882-19efe5a1c80c","Type":"ContainerStarted","Data":"b45eaf2b22d7276a635cd3590a06e2be111b16e4117790ebfaaeb36e5f4708b9"}
Oct 30 03:15:40.406512 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.406439    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.406424689 podStartE2EDuration="1.406424689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:40.404877115 +0000 UTC m=+2019.411467368" watchObservedRunningTime="2026-10-30 03:15:40.406424689 +0000 UTC m=+2019.413014942"
Oct 30 03:15:40.490121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.490086    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:40.614787 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.614715    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-rbac\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.614787 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.614746    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-conf\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.614787 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.614785    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.615039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.614829    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.615039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.614886    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-tenants\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.615039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.614924    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-ca-bundle\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.615039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615024    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.615237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615080    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-mtls\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.615237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615091    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-rbac" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:40.615237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615120    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27060c3-32c9-4132-b8d2-2aa776a69d12-kube-api-access-nlt9q\" (UniqueName: \"kubernetes.io/projected/f27060c3-32c9-4132-b8d2-2aa776a69d12-kube-api-access-nlt9q\") pod \"f27060c3-32c9-4132-b8d2-2aa776a69d12\" (UID: \"f27060c3-32c9-4132-b8d2-2aa776a69d12\") "
Oct 30 03:15:40.615395 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615232    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-conf" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:40.615395 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615322    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-serving-cabundle" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:40.615619 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615600    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.615711 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615624    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.615711 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615639    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.615824 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.615794    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-ca-bundle" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:40.617424 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.617399    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27060c3-32c9-4132-b8d2-2aa776a69d12-kube-api-access-nlt9q" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "kube-api-access-nlt9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:40.617613 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.617578    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-mtls" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:40.617613 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.617591    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:40.617758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.617681    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-tenants" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:40.618052 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.618025    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "f27060c3-32c9-4132-b8d2-2aa776a69d12" (UID: "f27060c3-32c9-4132-b8d2-2aa776a69d12"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:15:40.716916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.716887    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-gateway-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.716916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.716911    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.717122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.716942    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:15:40.717122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.716952    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.717122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.716962    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlt9q\" (UniqueName: \"kubernetes.io/projected/f27060c3-32c9-4132-b8d2-2aa776a69d12-kube-api-access-nlt9q\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.717122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.716971    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f27060c3-32c9-4132-b8d2-2aa776a69d12-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:40.727582 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.727561    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:15:40.818346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:40.818319    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\""
Oct 30 03:15:41.158898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:41.158867    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2127890e-19db-4608-9e64-36573a71ee02" path="/var/lib/kubelet/pods/2127890e-19db-4608-9e64-36573a71ee02/volumes"
Oct 30 03:15:41.372072 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:41.372038    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:15:41.372072 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:41.372058    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f27060c3-32c9-4132-b8d2-2aa776a69d12","Type":"ContainerDied","Data":"827800f665b1d549b530a65f5d8de57359c3d0b9000229f30ebaf85063e7b5a6"}
Oct 30 03:15:41.372588 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:41.372124    4334 scope.go:122] "RemoveContainer" containerID="0084a33835c4b5a0b80cc6ec9046fb5edfa45ba7c0656b9cd831b8bab2de5f8c"
Oct 30 03:15:41.380418 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:41.380399    4334 scope.go:122] "RemoveContainer" containerID="a5003a535af68c251ac57737068f7521f2c62e4a2aaa809cd480a725784d8b07"
Oct 30 03:15:41.387993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:41.387978    4334 scope.go:122] "RemoveContainer" containerID="336babf49a9ebecae200e1a296e14f3ab6b9fc6cae93a7cd8d57fb0d82f4687a"
Oct 30 03:15:41.431695 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:41.431642    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:41.442936 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:41.442910    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:15:43.159180 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:43.159115    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" path="/var/lib/kubelet/pods/f27060c3-32c9-4132-b8d2-2aa776a69d12/volumes"
Oct 30 03:15:46.519669 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.519629    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:46.520130 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.519891    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="89727543-0048-4117-9882-19efe5a1c80c" containerName="otc-container" containerID="cri-o://8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12" gracePeriod=30
Oct 30 03:15:46.765691 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.765666    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:46.772747 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.772515    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-data-science-collector-ta-client-cert\") pod \"89727543-0048-4117-9882-19efe5a1c80c\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") "
Oct 30 03:15:46.772747 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.772740    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-tls-certs\") pod \"89727543-0048-4117-9882-19efe5a1c80c\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") "
Oct 30 03:15:46.772934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.772850    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-monitoring-tls-certs\") pod \"89727543-0048-4117-9882-19efe5a1c80c\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") "
Oct 30 03:15:46.772934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.772888    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89727543-0048-4117-9882-19efe5a1c80c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/89727543-0048-4117-9882-19efe5a1c80c-otc-internal\") pod \"89727543-0048-4117-9882-19efe5a1c80c\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") "
Oct 30 03:15:46.772934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.772918    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89727543-0048-4117-9882-19efe5a1c80c-kube-api-access-xgf46\" (UniqueName: \"kubernetes.io/projected/89727543-0048-4117-9882-19efe5a1c80c-kube-api-access-xgf46\") pod \"89727543-0048-4117-9882-19efe5a1c80c\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") "
Oct 30 03:15:46.773358 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.773328    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89727543-0048-4117-9882-19efe5a1c80c-otc-internal" pod "89727543-0048-4117-9882-19efe5a1c80c" (UID: "89727543-0048-4117-9882-19efe5a1c80c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:46.774862 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.774831    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-data-science-collector-ta-client-cert" pod "89727543-0048-4117-9882-19efe5a1c80c" (UID: "89727543-0048-4117-9882-19efe5a1c80c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:46.774862 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.774852    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-tls-certs" pod "89727543-0048-4117-9882-19efe5a1c80c" (UID: "89727543-0048-4117-9882-19efe5a1c80c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:46.774981 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.774895    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-monitoring-tls-certs" pod "89727543-0048-4117-9882-19efe5a1c80c" (UID: "89727543-0048-4117-9882-19efe5a1c80c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:46.775014 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.774977    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89727543-0048-4117-9882-19efe5a1c80c-kube-api-access-xgf46" pod "89727543-0048-4117-9882-19efe5a1c80c" (UID: "89727543-0048-4117-9882-19efe5a1c80c"). InnerVolumeSpecName "kube-api-access-xgf46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:46.874176 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.874139    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-otlp-tls-certs\") pod \"89727543-0048-4117-9882-19efe5a1c80c\" (UID: \"89727543-0048-4117-9882-19efe5a1c80c\") "
Oct 30 03:15:46.874372 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.874360    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:46.874424 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.874375    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/89727543-0048-4117-9882-19efe5a1c80c-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:46.874424 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.874385    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgf46\" (UniqueName: \"kubernetes.io/projected/89727543-0048-4117-9882-19efe5a1c80c-kube-api-access-xgf46\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:46.874424 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.874402    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:46.874553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.874424    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:46.876158 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.876135    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-otlp-tls-certs" pod "89727543-0048-4117-9882-19efe5a1c80c" (UID: "89727543-0048-4117-9882-19efe5a1c80c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:46.975790 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:46.975762    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/89727543-0048-4117-9882-19efe5a1c80c-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:47.408896 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.408864    4334 generic.go:356] "Generic (PLEG): container finished" podID="89727543-0048-4117-9882-19efe5a1c80c" containerID="8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12" exitCode=0
Oct 30 03:15:47.409042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.408912    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"89727543-0048-4117-9882-19efe5a1c80c","Type":"ContainerDied","Data":"8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12"}
Oct 30 03:15:47.409042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.408926    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.409042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.408951    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"89727543-0048-4117-9882-19efe5a1c80c","Type":"ContainerDied","Data":"b45eaf2b22d7276a635cd3590a06e2be111b16e4117790ebfaaeb36e5f4708b9"}
Oct 30 03:15:47.409042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.408973    4334 scope.go:122] "RemoveContainer" containerID="8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12"
Oct 30 03:15:47.417602 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.417587    4334 scope.go:122] "RemoveContainer" containerID="8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12"
Oct 30 03:15:47.417881 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:15:47.417861    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12\": container with ID starting with 8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12 not found: ID does not exist" containerID="8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12"
Oct 30 03:15:47.417916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.417890    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12"} err="failed to get container status \"8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12\": rpc error: code = NotFound desc = could not find container \"8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12\": container with ID starting with 8f30a544da32f2e94ea0c629bc085bf7c57e0f5995e5309f82a1e75e9966ae12 not found: ID does not exist"
Oct 30 03:15:47.455611 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.455559    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:47.466767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.466738    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:47.497172 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497148    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:47.497567 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497551    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway"
Oct 30 03:15:47.497645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497569    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway"
Oct 30 03:15:47.497645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497585    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo"
Oct 30 03:15:47.497645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497593    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo"
Oct 30 03:15:47.497645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497606    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway-opa"
Oct 30 03:15:47.497645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497613    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway-opa"
Oct 30 03:15:47.497645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497637    4334 cpu_manager.go:414] "containerMap: removing container" podUID="89727543-0048-4117-9882-19efe5a1c80c" containerName="otc-container"
Oct 30 03:15:47.497645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497644    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89727543-0048-4117-9882-19efe5a1c80c" containerName="otc-container"
Oct 30 03:15:47.497967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497736    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway"
Oct 30 03:15:47.497967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497780    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo"
Oct 30 03:15:47.497967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497791    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27060c3-32c9-4132-b8d2-2aa776a69d12" containerName="tempo-gateway-opa"
Oct 30 03:15:47.497967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.497804    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="89727543-0048-4117-9882-19efe5a1c80c" containerName="otc-container"
Oct 30 03:15:47.502399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.500631    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.516657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.516628    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:47.516775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.516687    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:47.516775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.516761    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:47.516885 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.516866    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-770491e3\""
Oct 30 03:15:47.517004 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.516956    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:47.517004 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.516996    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:47.517232 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.517213    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:47.581631 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.581608    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8723465b-624c-45b6-b947-ff97e3148613-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.581939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.581656    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.581939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.581680    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.581939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.581782    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.581939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.581815    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.581939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.581841    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl9tb\" (UniqueName: \"kubernetes.io/projected/8723465b-624c-45b6-b947-ff97e3148613-kube-api-access-bl9tb\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.682657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.682630    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.682763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.682664    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.682763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.682687    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl9tb\" (UniqueName: \"kubernetes.io/projected/8723465b-624c-45b6-b947-ff97e3148613-kube-api-access-bl9tb\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.682851 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.682816    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8723465b-624c-45b6-b947-ff97e3148613-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.682934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.682919    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.682972 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.682962    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.683532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.683508    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8723465b-624c-45b6-b947-ff97e3148613-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.685216 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.685195    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.685312 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.685228    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.685385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.685363    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.685461 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.685414    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.700484 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.700464    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl9tb\" (UniqueName: \"kubernetes.io/projected/8723465b-624c-45b6-b947-ff97e3148613-kube-api-access-bl9tb\") pod \"data-science-collector-collector-1\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:47.812866 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:47.812807    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:48.109340 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:48.109262    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:48.112890 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:15:48.112861    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8723465b_624c_45b6_b947_ff97e3148613.slice/crio-0b68e4c327d3f6bbfdcc7649e05e50d9d58e5da3076a2e5f5b09fc04cd2cccf0 WatchSource:0}: Error finding container 0b68e4c327d3f6bbfdcc7649e05e50d9d58e5da3076a2e5f5b09fc04cd2cccf0: Status 404 returned error can't find the container with id 0b68e4c327d3f6bbfdcc7649e05e50d9d58e5da3076a2e5f5b09fc04cd2cccf0
Oct 30 03:15:48.115533 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:48.115499    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:48.115620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:48.115571    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:48.416060 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:48.416024    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8723465b-624c-45b6-b947-ff97e3148613","Type":"ContainerStarted","Data":"d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460"}
Oct 30 03:15:48.416236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:48.416095    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8723465b-624c-45b6-b947-ff97e3148613","Type":"ContainerStarted","Data":"0b68e4c327d3f6bbfdcc7649e05e50d9d58e5da3076a2e5f5b09fc04cd2cccf0"}
Oct 30 03:15:48.474401 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:48.474350    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.474314725 podStartE2EDuration="1.474314725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:48.471617912 +0000 UTC m=+2027.478208168" watchObservedRunningTime="2026-10-30 03:15:48.474314725 +0000 UTC m=+2027.480904978"
Oct 30 03:15:49.158844 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:49.158805    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89727543-0048-4117-9882-19efe5a1c80c" path="/var/lib/kubelet/pods/89727543-0048-4117-9882-19efe5a1c80c/volumes"
Oct 30 03:15:50.978466 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:15:50.978408    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:15:54.833439 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:54.833396    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:54.834036 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:54.833655    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="8723465b-624c-45b6-b947-ff97e3148613" containerName="otc-container" containerID="cri-o://d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460" gracePeriod=30
Oct 30 03:15:55.089146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.089094    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.145018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.144994    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-tls-certs\") pod \"8723465b-624c-45b6-b947-ff97e3148613\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") "
Oct 30 03:15:55.145195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.145029    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-data-science-collector-ta-client-cert\") pod \"8723465b-624c-45b6-b947-ff97e3148613\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") "
Oct 30 03:15:55.145195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.145054    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-monitoring-tls-certs\") pod \"8723465b-624c-45b6-b947-ff97e3148613\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") "
Oct 30 03:15:55.145195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.145110    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8723465b-624c-45b6-b947-ff97e3148613-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8723465b-624c-45b6-b947-ff97e3148613-otc-internal\") pod \"8723465b-624c-45b6-b947-ff97e3148613\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") "
Oct 30 03:15:55.145195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.145127    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8723465b-624c-45b6-b947-ff97e3148613-kube-api-access-bl9tb\" (UniqueName: \"kubernetes.io/projected/8723465b-624c-45b6-b947-ff97e3148613-kube-api-access-bl9tb\") pod \"8723465b-624c-45b6-b947-ff97e3148613\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") "
Oct 30 03:15:55.145195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.145175    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-otlp-tls-certs\") pod \"8723465b-624c-45b6-b947-ff97e3148613\" (UID: \"8723465b-624c-45b6-b947-ff97e3148613\") "
Oct 30 03:15:55.145590 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.145552    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8723465b-624c-45b6-b947-ff97e3148613-otc-internal" pod "8723465b-624c-45b6-b947-ff97e3148613" (UID: "8723465b-624c-45b6-b947-ff97e3148613"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:55.147328 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.147301    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-monitoring-tls-certs" pod "8723465b-624c-45b6-b947-ff97e3148613" (UID: "8723465b-624c-45b6-b947-ff97e3148613"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:55.147498 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.147473    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-otlp-tls-certs" pod "8723465b-624c-45b6-b947-ff97e3148613" (UID: "8723465b-624c-45b6-b947-ff97e3148613"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:55.147815 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.147773    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-tls-certs" pod "8723465b-624c-45b6-b947-ff97e3148613" (UID: "8723465b-624c-45b6-b947-ff97e3148613"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:55.147904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.147855    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-data-science-collector-ta-client-cert" pod "8723465b-624c-45b6-b947-ff97e3148613" (UID: "8723465b-624c-45b6-b947-ff97e3148613"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:55.148147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.148127    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8723465b-624c-45b6-b947-ff97e3148613-kube-api-access-bl9tb" pod "8723465b-624c-45b6-b947-ff97e3148613" (UID: "8723465b-624c-45b6-b947-ff97e3148613"). InnerVolumeSpecName "kube-api-access-bl9tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:55.246189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.246159    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8723465b-624c-45b6-b947-ff97e3148613-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.246189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.246184    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl9tb\" (UniqueName: \"kubernetes.io/projected/8723465b-624c-45b6-b947-ff97e3148613-kube-api-access-bl9tb\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.246337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.246197    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.246337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.246209    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.246337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.246221    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.246337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.246233    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8723465b-624c-45b6-b947-ff97e3148613-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:15:55.457991 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.457963    4334 generic.go:356] "Generic (PLEG): container finished" podID="8723465b-624c-45b6-b947-ff97e3148613" containerID="d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460" exitCode=0
Oct 30 03:15:55.458135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.458009    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8723465b-624c-45b6-b947-ff97e3148613","Type":"ContainerDied","Data":"d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460"}
Oct 30 03:15:55.458135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.458012    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.458135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.458052    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8723465b-624c-45b6-b947-ff97e3148613","Type":"ContainerDied","Data":"0b68e4c327d3f6bbfdcc7649e05e50d9d58e5da3076a2e5f5b09fc04cd2cccf0"}
Oct 30 03:15:55.458135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.458073    4334 scope.go:122] "RemoveContainer" containerID="d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460"
Oct 30 03:15:55.466669 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.466648    4334 scope.go:122] "RemoveContainer" containerID="d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460"
Oct 30 03:15:55.466940 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:15:55.466921    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460\": container with ID starting with d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460 not found: ID does not exist" containerID="d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460"
Oct 30 03:15:55.467023 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.466951    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460"} err="failed to get container status \"d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460\": rpc error: code = NotFound desc = could not find container \"d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460\": container with ID starting with d7ec7949c8da76175e7c7f1419e7cffb356646208766db0c0abb0eb50764a460 not found: ID does not exist"
Oct 30 03:15:55.506476 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.506433    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:55.517676 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.517656    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:55.548264 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.548244    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:55.548720 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.548703    4334 cpu_manager.go:414] "containerMap: removing container" podUID="8723465b-624c-45b6-b947-ff97e3148613" containerName="otc-container"
Oct 30 03:15:55.548819 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.548721    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8723465b-624c-45b6-b947-ff97e3148613" containerName="otc-container"
Oct 30 03:15:55.548877 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.548834    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="8723465b-624c-45b6-b947-ff97e3148613" containerName="otc-container"
Oct 30 03:15:55.553090 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.553073    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.567873 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.567843    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:15:55.567998 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.567969    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:15:55.568060 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.568036    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:15:55.568183 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.568167    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:15:55.568440 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.568423    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:15:55.568967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.568947    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:55.582229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.582210    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:15:55.649079 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.649050    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.649079 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.649080    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9lq\" (UniqueName: \"kubernetes.io/projected/b5f764c8-34e4-47f2-b79b-5d883a137597-kube-api-access-hh9lq\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.649237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.649106    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.649237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.649213    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.649311 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.649254    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5f764c8-34e4-47f2-b79b-5d883a137597-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.649311 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.649274    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.750134 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.750065    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.750134 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.750114    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5f764c8-34e4-47f2-b79b-5d883a137597-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.750322 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.750219    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.750322 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.750262    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.750322 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.750280    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9lq\" (UniqueName: \"kubernetes.io/projected/b5f764c8-34e4-47f2-b79b-5d883a137597-kube-api-access-hh9lq\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.750506 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.750328    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.750870 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.750839    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5f764c8-34e4-47f2-b79b-5d883a137597-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.752763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.752739    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.752862 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.752766    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.753066 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.753048    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.753125 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.753069    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.768993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.768975    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9lq\" (UniqueName: \"kubernetes.io/projected/b5f764c8-34e4-47f2-b79b-5d883a137597-kube-api-access-hh9lq\") pod \"data-science-collector-collector-1\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:55.865386 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:55.865358    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:15:56.162103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:56.162069    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:15:56.167574 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:15:56.167548    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f764c8_34e4_47f2_b79b_5d883a137597.slice/crio-4e2d91a940d571e2081911ea34e6f71f098d6e51b742b91d0d5f7699007e6e5b WatchSource:0}: Error finding container 4e2d91a940d571e2081911ea34e6f71f098d6e51b742b91d0d5f7699007e6e5b: Status 404 returned error can't find the container with id 4e2d91a940d571e2081911ea34e6f71f098d6e51b742b91d0d5f7699007e6e5b
Oct 30 03:15:56.169777 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:56.169743    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:56.169875 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:56.169802    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:15:56.465710 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:56.465628    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b5f764c8-34e4-47f2-b79b-5d883a137597","Type":"ContainerStarted","Data":"3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d"}
Oct 30 03:15:56.465710 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:56.465680    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b5f764c8-34e4-47f2-b79b-5d883a137597","Type":"ContainerStarted","Data":"4e2d91a940d571e2081911ea34e6f71f098d6e51b742b91d0d5f7699007e6e5b"}
Oct 30 03:15:56.503327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:56.503230    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.503213407 podStartE2EDuration="1.503213407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:56.502954586 +0000 UTC m=+2035.509544841" watchObservedRunningTime="2026-10-30 03:15:56.503213407 +0000 UTC m=+2035.509803665"
Oct 30 03:15:57.158370 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:15:57.158337    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8723465b-624c-45b6-b947-ff97e3148613" path="/var/lib/kubelet/pods/8723465b-624c-45b6-b947-ff97e3148613/volumes"
Oct 30 03:16:03.455070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.455030    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:03.455518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.455223    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b5f764c8-34e4-47f2-b79b-5d883a137597" containerName="otc-container" containerID="cri-o://3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d" gracePeriod=30
Oct 30 03:16:03.701258 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.701236    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:03.823579 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.823497    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f764c8-34e4-47f2-b79b-5d883a137597-kube-api-access-hh9lq\" (UniqueName: \"kubernetes.io/projected/b5f764c8-34e4-47f2-b79b-5d883a137597-kube-api-access-hh9lq\") pod \"b5f764c8-34e4-47f2-b79b-5d883a137597\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") "
Oct 30 03:16:03.823579 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.823534    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-tls-certs\") pod \"b5f764c8-34e4-47f2-b79b-5d883a137597\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") "
Oct 30 03:16:03.823808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.823595    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f764c8-34e4-47f2-b79b-5d883a137597-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5f764c8-34e4-47f2-b79b-5d883a137597-otc-internal\") pod \"b5f764c8-34e4-47f2-b79b-5d883a137597\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") "
Oct 30 03:16:03.823808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.823632    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-data-science-collector-ta-client-cert\") pod \"b5f764c8-34e4-47f2-b79b-5d883a137597\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") "
Oct 30 03:16:03.823808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.823650    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-monitoring-tls-certs\") pod \"b5f764c8-34e4-47f2-b79b-5d883a137597\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") "
Oct 30 03:16:03.823808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.823671    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-otlp-tls-certs\") pod \"b5f764c8-34e4-47f2-b79b-5d883a137597\" (UID: \"b5f764c8-34e4-47f2-b79b-5d883a137597\") "
Oct 30 03:16:03.824176 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.824148    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f764c8-34e4-47f2-b79b-5d883a137597-otc-internal" pod "b5f764c8-34e4-47f2-b79b-5d883a137597" (UID: "b5f764c8-34e4-47f2-b79b-5d883a137597"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:03.825922 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.825897    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-data-science-collector-ta-client-cert" pod "b5f764c8-34e4-47f2-b79b-5d883a137597" (UID: "b5f764c8-34e4-47f2-b79b-5d883a137597"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:03.826345 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.826321    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-monitoring-tls-certs" pod "b5f764c8-34e4-47f2-b79b-5d883a137597" (UID: "b5f764c8-34e4-47f2-b79b-5d883a137597"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:03.826496 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.826476    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f764c8-34e4-47f2-b79b-5d883a137597-kube-api-access-hh9lq" pod "b5f764c8-34e4-47f2-b79b-5d883a137597" (UID: "b5f764c8-34e4-47f2-b79b-5d883a137597"). InnerVolumeSpecName "kube-api-access-hh9lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:03.826620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.826601    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-otlp-tls-certs" pod "b5f764c8-34e4-47f2-b79b-5d883a137597" (UID: "b5f764c8-34e4-47f2-b79b-5d883a137597"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:03.827570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.827546    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-tls-certs" pod "b5f764c8-34e4-47f2-b79b-5d883a137597" (UID: "b5f764c8-34e4-47f2-b79b-5d883a137597"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:03.925013 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.924983    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5f764c8-34e4-47f2-b79b-5d883a137597-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:03.925013 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.925009    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:03.925013 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.925019    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:03.925215 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.925028    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:03.925215 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.925036    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh9lq\" (UniqueName: \"kubernetes.io/projected/b5f764c8-34e4-47f2-b79b-5d883a137597-kube-api-access-hh9lq\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:03.925215 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:03.925050    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f764c8-34e4-47f2-b79b-5d883a137597-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:04.507226 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.507192    4334 generic.go:356] "Generic (PLEG): container finished" podID="b5f764c8-34e4-47f2-b79b-5d883a137597" containerID="3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d" exitCode=0
Oct 30 03:16:04.507675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.507250    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.507675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.507290    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b5f764c8-34e4-47f2-b79b-5d883a137597","Type":"ContainerDied","Data":"3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d"}
Oct 30 03:16:04.507675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.507346    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b5f764c8-34e4-47f2-b79b-5d883a137597","Type":"ContainerDied","Data":"4e2d91a940d571e2081911ea34e6f71f098d6e51b742b91d0d5f7699007e6e5b"}
Oct 30 03:16:04.507675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.507369    4334 scope.go:122] "RemoveContainer" containerID="3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d"
Oct 30 03:16:04.516542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.516525    4334 scope.go:122] "RemoveContainer" containerID="3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d"
Oct 30 03:16:04.516781 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:04.516758    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d\": container with ID starting with 3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d not found: ID does not exist" containerID="3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d"
Oct 30 03:16:04.516852 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.516793    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d"} err="failed to get container status \"3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d\": rpc error: code = NotFound desc = could not find container \"3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d\": container with ID starting with 3c7388f813ca78e016eb2de62148384c14dceb001c06c9fcf237e90cf5f1919d not found: ID does not exist"
Oct 30 03:16:04.557412 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.557388    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:04.569053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.569032    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:04.596591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.596567    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:04.596929 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.596916    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b5f764c8-34e4-47f2-b79b-5d883a137597" containerName="otc-container"
Oct 30 03:16:04.596981 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.596930    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f764c8-34e4-47f2-b79b-5d883a137597" containerName="otc-container"
Oct 30 03:16:04.597022 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.596987    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f764c8-34e4-47f2-b79b-5d883a137597" containerName="otc-container"
Oct 30 03:16:04.601839 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.601816    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.618757 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.618736    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:04.618931 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.618736    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:04.618931 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.618870    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:04.619074 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.618945    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:04.620933 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.620909    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:04.629108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.629088    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-50fd48e3\""
Oct 30 03:16:04.629200 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.629152    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:04.730323 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.730278    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.730323 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.730327    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.730550 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.730352    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.730550 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.730474    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.730550 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.730535    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1c562276-772b-4bd3-a634-9cafb54fbe94-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.730731 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.730587    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2nmk\" (UniqueName: \"kubernetes.io/projected/1c562276-772b-4bd3-a634-9cafb54fbe94-kube-api-access-z2nmk\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.831581 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.831507    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.831581 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.831548    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.831804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.831621    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1c562276-772b-4bd3-a634-9cafb54fbe94-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.831804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.831644    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2nmk\" (UniqueName: \"kubernetes.io/projected/1c562276-772b-4bd3-a634-9cafb54fbe94-kube-api-access-z2nmk\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.831804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.831706    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.831804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.831740    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.832774 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.832748    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1c562276-772b-4bd3-a634-9cafb54fbe94-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.834794 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.834760    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.835478 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.835441    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.837463 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.837426    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.839180 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.839157    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.847149 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.847084    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:16:04.851610 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.851591    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.852948 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.852924    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2nmk\" (UniqueName: \"kubernetes.io/projected/1c562276-772b-4bd3-a634-9cafb54fbe94-kube-api-access-z2nmk\") pod \"data-science-collector-collector-1\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.865814 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.865726    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:16:04.865814 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.865752    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:16:04.865814 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.865760    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:16:04.865814 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.865789    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:16:04.866076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.865854    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:16:04.868745 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.868728    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-zb426\""
Oct 30 03:16:04.872031 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.872012    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:16:04.914268 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.914249    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:04.932204 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.932180    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.932303 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.932243    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.932380 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.932359    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5j6r\" (UniqueName: \"kubernetes.io/projected/7498af25-0e73-49de-acad-64efeff7f900-kube-api-access-x5j6r\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.932428 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.932401    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.932576 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.932556    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.932633 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.932592    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.932698 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.932642    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.932777 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.932743    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:04.965949 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.965891    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:16:04.965949 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:04.965935    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:16:05.033813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.033781    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.033948 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.033827    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.033948 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.033857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5j6r\" (UniqueName: \"kubernetes.io/projected/7498af25-0e73-49de-acad-64efeff7f900-kube-api-access-x5j6r\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.033948 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.033888    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.034076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.033958    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.034076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.033986    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.034076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.034014    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.034076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.034042    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.034636 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.034609    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.034999 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.034975    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.034999 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.035006    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.035195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.035008    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.036661 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.036637    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.036736 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.036677    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.037278 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.037259    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.052641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.052623    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5j6r\" (UniqueName: \"kubernetes.io/projected/7498af25-0e73-49de-acad-64efeff7f900-kube-api-access-x5j6r\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:05.159332 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.159304    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f764c8-34e4-47f2-b79b-5d883a137597" path="/var/lib/kubelet/pods/b5f764c8-34e4-47f2-b79b-5d883a137597/volumes"
Oct 30 03:16:05.208658 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.208616    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:05.212669 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:16:05.212637    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c562276_772b_4bd3_a634_9cafb54fbe94.slice/crio-fc26aa14a194f1fedd8eccbade55a6a9ea80bbbe2bc1c85357375f38ab766e68 WatchSource:0}: Error finding container fc26aa14a194f1fedd8eccbade55a6a9ea80bbbe2bc1c85357375f38ab766e68: Status 404 returned error can't find the container with id fc26aa14a194f1fedd8eccbade55a6a9ea80bbbe2bc1c85357375f38ab766e68
Oct 30 03:16:05.214763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.214735    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:05.214835 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.214792    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:05.513972 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.513890    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1c562276-772b-4bd3-a634-9cafb54fbe94","Type":"ContainerStarted","Data":"3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24"}
Oct 30 03:16:05.513972 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.513955    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1c562276-772b-4bd3-a634-9cafb54fbe94","Type":"ContainerStarted","Data":"fc26aa14a194f1fedd8eccbade55a6a9ea80bbbe2bc1c85357375f38ab766e68"}
Oct 30 03:16:05.568498 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:05.568416    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.568398611 podStartE2EDuration="1.568398611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:05.566763908 +0000 UTC m=+2044.573354163" watchObservedRunningTime="2026-10-30 03:16:05.568398611 +0000 UTC m=+2044.574988866"
Oct 30 03:16:12.797940 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:12.797911    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:12.824553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:12.824528    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:12.898827 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:12.898794    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:12.898955 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:12.898928    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:12.912122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:12.912094    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:12.912554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:12.912506    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:12.912773 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:12.912746    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1c562276-772b-4bd3-a634-9cafb54fbe94" containerName="otc-container" containerID="cri-o://3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24" gracePeriod=30
Oct 30 03:16:13.019298 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.019260    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:13.047286 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.047256    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:13.168797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.168767    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.281909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.281875    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:13.302050 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.302022    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-otlp-tls-certs\") pod \"1c562276-772b-4bd3-a634-9cafb54fbe94\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") "
Oct 30 03:16:13.302152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.302066    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c562276-772b-4bd3-a634-9cafb54fbe94-kube-api-access-z2nmk\" (UniqueName: \"kubernetes.io/projected/1c562276-772b-4bd3-a634-9cafb54fbe94-kube-api-access-z2nmk\") pod \"1c562276-772b-4bd3-a634-9cafb54fbe94\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") "
Oct 30 03:16:13.302152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.302107    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c562276-772b-4bd3-a634-9cafb54fbe94-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1c562276-772b-4bd3-a634-9cafb54fbe94-otc-internal\") pod \"1c562276-772b-4bd3-a634-9cafb54fbe94\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") "
Oct 30 03:16:13.302152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.302129    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-tls-certs\") pod \"1c562276-772b-4bd3-a634-9cafb54fbe94\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") "
Oct 30 03:16:13.302299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.302170    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-monitoring-tls-certs\") pod \"1c562276-772b-4bd3-a634-9cafb54fbe94\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") "
Oct 30 03:16:13.302299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.302241    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-data-science-collector-ta-client-cert\") pod \"1c562276-772b-4bd3-a634-9cafb54fbe94\" (UID: \"1c562276-772b-4bd3-a634-9cafb54fbe94\") "
Oct 30 03:16:13.302759 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.302540    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c562276-772b-4bd3-a634-9cafb54fbe94-otc-internal" pod "1c562276-772b-4bd3-a634-9cafb54fbe94" (UID: "1c562276-772b-4bd3-a634-9cafb54fbe94"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:13.304350 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.304327    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-monitoring-tls-certs" pod "1c562276-772b-4bd3-a634-9cafb54fbe94" (UID: "1c562276-772b-4bd3-a634-9cafb54fbe94"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:13.304515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.304410    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c562276-772b-4bd3-a634-9cafb54fbe94-kube-api-access-z2nmk" pod "1c562276-772b-4bd3-a634-9cafb54fbe94" (UID: "1c562276-772b-4bd3-a634-9cafb54fbe94"). InnerVolumeSpecName "kube-api-access-z2nmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:13.304606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.304589    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-otlp-tls-certs" pod "1c562276-772b-4bd3-a634-9cafb54fbe94" (UID: "1c562276-772b-4bd3-a634-9cafb54fbe94"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:13.304811 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.304786    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-data-science-collector-ta-client-cert" pod "1c562276-772b-4bd3-a634-9cafb54fbe94" (UID: "1c562276-772b-4bd3-a634-9cafb54fbe94"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:13.304989 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.304965    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-tls-certs" pod "1c562276-772b-4bd3-a634-9cafb54fbe94" (UID: "1c562276-772b-4bd3-a634-9cafb54fbe94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:13.403227 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.403195    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:13.403227 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.403224    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2nmk\" (UniqueName: \"kubernetes.io/projected/1c562276-772b-4bd3-a634-9cafb54fbe94-kube-api-access-z2nmk\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:13.403227 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.403235    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1c562276-772b-4bd3-a634-9cafb54fbe94-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:13.403475 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.403244    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:13.403475 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.403253    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:13.403475 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.403265    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1c562276-772b-4bd3-a634-9cafb54fbe94-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:13.561004 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.560966    4334 generic.go:356] "Generic (PLEG): container finished" podID="1c562276-772b-4bd3-a634-9cafb54fbe94" containerID="3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24" exitCode=0
Oct 30 03:16:13.561188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.561023    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.561188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.561026    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1c562276-772b-4bd3-a634-9cafb54fbe94","Type":"ContainerDied","Data":"3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24"}
Oct 30 03:16:13.561188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.561110    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1c562276-772b-4bd3-a634-9cafb54fbe94","Type":"ContainerDied","Data":"fc26aa14a194f1fedd8eccbade55a6a9ea80bbbe2bc1c85357375f38ab766e68"}
Oct 30 03:16:13.561188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.561143    4334 scope.go:122] "RemoveContainer" containerID="3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24"
Oct 30 03:16:13.570697 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.570680    4334 scope.go:122] "RemoveContainer" containerID="3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24"
Oct 30 03:16:13.570939 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:13.570919    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24\": container with ID starting with 3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24 not found: ID does not exist" containerID="3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24"
Oct 30 03:16:13.570983 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.570947    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24"} err="failed to get container status \"3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24\": rpc error: code = NotFound desc = could not find container \"3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24\": container with ID starting with 3369d33e7e58df3dfc8f1618e8ae207034611ac48c3f681f7d84af9ddc641f24 not found: ID does not exist"
Oct 30 03:16:13.609022 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.608962    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:13.611188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.611161    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:16:13.619861 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.619841    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:13.626819 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:16:13.626795    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7498af25_0e73_49de_acad_64efeff7f900.slice/crio-942ab7d2a0bf85986d5c6c742080db023ff24b80a834001d362a5153d069aaa9 WatchSource:0}: Error finding container 942ab7d2a0bf85986d5c6c742080db023ff24b80a834001d362a5153d069aaa9: Status 404 returned error can't find the container with id 942ab7d2a0bf85986d5c6c742080db023ff24b80a834001d362a5153d069aaa9
Oct 30 03:16:13.652309 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.652287    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:13.652805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.652787    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1c562276-772b-4bd3-a634-9cafb54fbe94" containerName="otc-container"
Oct 30 03:16:13.652805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.652806    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c562276-772b-4bd3-a634-9cafb54fbe94" containerName="otc-container"
Oct 30 03:16:13.652958 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.652878    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c562276-772b-4bd3-a634-9cafb54fbe94" containerName="otc-container"
Oct 30 03:16:13.657556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.657537    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.671880 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.671856    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:13.672576 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.672554    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:13.672665 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.672622    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:13.672724 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.672665    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:13.672724 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.672694    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:13.674846 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.674661    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:13.678527 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.678508    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:16:13.705119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.705095    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.705228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.705148    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.705228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.705195    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.705345 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.705327    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.705469 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.705360    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.705544 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.705436    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkg8d\" (UniqueName: \"kubernetes.io/projected/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-kube-api-access-hkg8d\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.807111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.807026    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkg8d\" (UniqueName: \"kubernetes.io/projected/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-kube-api-access-hkg8d\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.807507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.807141    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.807507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.807188    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.807507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.807237    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.807507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.807295    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.807507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.807324    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.808710 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.808687    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.810512 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.810291    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.810617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.810599    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.810738 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.810719    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.811399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.811381    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.826044 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.826023    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkg8d\" (UniqueName: \"kubernetes.io/projected/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-kube-api-access-hkg8d\") pod \"data-science-collector-collector-1\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:13.953675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.953641    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:16:13.970060 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:13.970034    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:14.265347 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.265272    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:14.269723 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:16:14.269694    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60c3004_9c03_49ca_b1e7_4d7a92c983f6.slice/crio-2ab0314841d4adebc1e4bca58fa4e0eec0988051b882f764c2b372a7385d6b27 WatchSource:0}: Error finding container 2ab0314841d4adebc1e4bca58fa4e0eec0988051b882f764c2b372a7385d6b27: Status 404 returned error can't find the container with id 2ab0314841d4adebc1e4bca58fa4e0eec0988051b882f764c2b372a7385d6b27
Oct 30 03:16:14.272416 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.272386    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:14.272527 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.272442    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:14.573716 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.573629    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a60c3004-9c03-49ca-b1e7-4d7a92c983f6","Type":"ContainerStarted","Data":"bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e"}
Oct 30 03:16:14.573716 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.573689    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a60c3004-9c03-49ca-b1e7-4d7a92c983f6","Type":"ContainerStarted","Data":"2ab0314841d4adebc1e4bca58fa4e0eec0988051b882f764c2b372a7385d6b27"}
Oct 30 03:16:14.575734 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.575707    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"7498af25-0e73-49de-acad-64efeff7f900","Type":"ContainerStarted","Data":"16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7"}
Oct 30 03:16:14.575855 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.575750    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"7498af25-0e73-49de-acad-64efeff7f900","Type":"ContainerStarted","Data":"85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721"}
Oct 30 03:16:14.575855 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.575767    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"7498af25-0e73-49de-acad-64efeff7f900","Type":"ContainerStarted","Data":"0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f"}
Oct 30 03:16:14.575855 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.575782    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"7498af25-0e73-49de-acad-64efeff7f900","Type":"ContainerStarted","Data":"942ab7d2a0bf85986d5c6c742080db023ff24b80a834001d362a5153d069aaa9"}
Oct 30 03:16:14.576021 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.575922    4334 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" secret="" err="secret \"tempo-data-science-tempomonolithic-dockercfg-zb426\" not found"
Oct 30 03:16:14.576021 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.576007    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:14.576122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.576035    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:14.576122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.576057    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:14.578484 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.578442    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:14.584586 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.584568    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:14.661755 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.661705    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6616887839999999 podStartE2EDuration="1.661688784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:14.612618449 +0000 UTC m=+2053.619208705" watchObservedRunningTime="2026-10-30 03:16:14.661688784 +0000 UTC m=+2053.668279039"
Oct 30 03:16:14.661965 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:14.661939    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=10.661930721000001 podStartE2EDuration="10.661930721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:14.66015919 +0000 UTC m=+2053.666749442" watchObservedRunningTime="2026-10-30 03:16:14.661930721 +0000 UTC m=+2053.668520975"
Oct 30 03:16:14.713527 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713495    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:14.713722 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713541    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:16:14.713722 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713567    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:15.213551143 +0000 UTC m=+2054.220141394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:14.713722 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713495    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:16:14.713722 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713596    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:15.213580102 +0000 UTC m=+2054.220170343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:16:14.713722 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713628    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:15.213614421 +0000 UTC m=+2054.220204657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:16:14.714000 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713748    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:16:14.714000 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713758    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:16:14.714000 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713767    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:16:14.714000 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713805    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:15.213793563 +0000 UTC m=+2054.220383796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:16:14.714000 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713823    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:15.213816693 +0000 UTC m=+2054.220406925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:16:14.714000 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713880    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:15.21386571 +0000 UTC m=+2054.220455945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:16:14.714000 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713901    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:14.714000 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:14.713937    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:15.213924929 +0000 UTC m=+2054.220515164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:15.158957 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:15.158926    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c562276-772b-4bd3-a634-9cafb54fbe94" path="/var/lib/kubelet/pods/1c562276-772b-4bd3-a634-9cafb54fbe94/volumes"
Oct 30 03:16:15.218947 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.218917    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:16:15.218947 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.218935    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:16:15.218947 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.218946    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.218935    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.218995    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:16.218974377 +0000 UTC m=+2055.225564612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.218943    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.219036    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.219045    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:16.219033195 +0000 UTC m=+2055.225623430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.219036    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.219101    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:16.219091049 +0000 UTC m=+2055.225681288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.219118    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:16.219110019 +0000 UTC m=+2055.225700252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.219133    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:16.21912471 +0000 UTC m=+2055.225714943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.219147    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:16.219139883 +0000 UTC m=+2055.225730116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:15.219160 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:15.219167    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:16.21915951 +0000 UTC m=+2055.225749742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:16:15.580390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:15.580360    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo" containerID="cri-o://0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f" gracePeriod=30
Oct 30 03:16:15.580572 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:15.580430    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo-gateway-opa" containerID="cri-o://16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7" gracePeriod=30
Oct 30 03:16:15.580572 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:15.580440    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo-gateway" containerID="cri-o://85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721" gracePeriod=30
Oct 30 03:16:16.228560 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228526    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:16:16.228560 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228546    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228604    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:18.228586081 +0000 UTC m=+2057.235176333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228608    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228624    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:18.228613804 +0000 UTC m=+2057.235204036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228644    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228671    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228650    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:18.22863403 +0000 UTC m=+2057.235224286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228673    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228716    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:18.228700854 +0000 UTC m=+2057.235291103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228733    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:18.228723336 +0000 UTC m=+2057.235313572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228734    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228749    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:18.228741055 +0000 UTC m=+2057.235331302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:16:16.229073 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:16.228795    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants podName:7498af25-0e73-49de-acad-64efeff7f900 nodeName:}" failed. No retries permitted until 2026-10-30 03:16:18.228778126 +0000 UTC m=+2057.235368376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "7498af25-0e73-49de-acad-64efeff7f900") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:16:16.587187 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:16.587106    4334 generic.go:356] "Generic (PLEG): container finished" podID="7498af25-0e73-49de-acad-64efeff7f900" containerID="16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7" exitCode=0
Oct 30 03:16:16.587187 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:16.587129    4334 generic.go:356] "Generic (PLEG): container finished" podID="7498af25-0e73-49de-acad-64efeff7f900" containerID="0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f" exitCode=0
Oct 30 03:16:16.587187 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:16.587174    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"7498af25-0e73-49de-acad-64efeff7f900","Type":"ContainerDied","Data":"16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7"}
Oct 30 03:16:16.587400 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:16.587223    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"7498af25-0e73-49de-acad-64efeff7f900","Type":"ContainerDied","Data":"0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f"}
Oct 30 03:16:16.934715 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:16.934685    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:17.036650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.036621    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.036804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.036741    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.036804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.036771    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.036804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.036795    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7498af25-0e73-49de-acad-64efeff7f900-kube-api-access-x5j6r\" (UniqueName: \"kubernetes.io/projected/7498af25-0e73-49de-acad-64efeff7f900-kube-api-access-x5j6r\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.036916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.036815    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.036916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.036910    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.037001 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.036944    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.037001 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.036987    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.037098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.037024    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle\") pod \"7498af25-0e73-49de-acad-64efeff7f900\" (UID: \"7498af25-0e73-49de-acad-64efeff7f900\") "
Oct 30 03:16:17.037098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.037033    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:17.037590 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.037321    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:17.037590 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.037388    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:17.037590 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.037512    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:17.037914 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.037893    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:17.039246 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.039222    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:17.039499 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.039468    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:17.039612 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.039494    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7498af25-0e73-49de-acad-64efeff7f900-kube-api-access-x5j6r" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "kube-api-access-x5j6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:17.039612 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.039579    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:17.039714 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.039631    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "7498af25-0e73-49de-acad-64efeff7f900" (UID: "7498af25-0e73-49de-acad-64efeff7f900"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:16:17.138310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.138251    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:17.138310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.138276    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:17.138310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.138287    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:17.138516 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.138314    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:16:17.138516 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.138326    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/7498af25-0e73-49de-acad-64efeff7f900-gateway-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:17.138516 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.138335    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5j6r\" (UniqueName: \"kubernetes.io/projected/7498af25-0e73-49de-acad-64efeff7f900-kube-api-access-x5j6r\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:17.138516 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.138352    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:17.138516 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.138361    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/7498af25-0e73-49de-acad-64efeff7f900-gateway-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:17.149701 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.149684    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:16:17.239344 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.239314    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\""
Oct 30 03:16:17.594613 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.594582    4334 generic.go:356] "Generic (PLEG): container finished" podID="7498af25-0e73-49de-acad-64efeff7f900" containerID="85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721" exitCode=0
Oct 30 03:16:17.594818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.594641    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:16:17.594818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.594666    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"7498af25-0e73-49de-acad-64efeff7f900","Type":"ContainerDied","Data":"85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721"}
Oct 30 03:16:17.594818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.594726    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"7498af25-0e73-49de-acad-64efeff7f900","Type":"ContainerDied","Data":"942ab7d2a0bf85986d5c6c742080db023ff24b80a834001d362a5153d069aaa9"}
Oct 30 03:16:17.594818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.594758    4334 scope.go:122] "RemoveContainer" containerID="16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7"
Oct 30 03:16:17.603034 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.603018    4334 scope.go:122] "RemoveContainer" containerID="85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721"
Oct 30 03:16:17.610507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.610486    4334 scope.go:122] "RemoveContainer" containerID="0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f"
Oct 30 03:16:17.618039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.618015    4334 scope.go:122] "RemoveContainer" containerID="16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7"
Oct 30 03:16:17.618301 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:17.618284    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7\": container with ID starting with 16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7 not found: ID does not exist" containerID="16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7"
Oct 30 03:16:17.618364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.618308    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7"} err="failed to get container status \"16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7\": rpc error: code = NotFound desc = could not find container \"16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7\": container with ID starting with 16b5634efb91c8b815ac638634fea16889d276d5711e734557a74b191ed596a7 not found: ID does not exist"
Oct 30 03:16:17.618364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.618327    4334 scope.go:122] "RemoveContainer" containerID="85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721"
Oct 30 03:16:17.618603 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:17.618587    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721\": container with ID starting with 85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721 not found: ID does not exist" containerID="85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721"
Oct 30 03:16:17.618652 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.618608    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721"} err="failed to get container status \"85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721\": rpc error: code = NotFound desc = could not find container \"85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721\": container with ID starting with 85dc689c70411a21f9d0191af09b2771bea78ebe2efa7ea595269dff3cccc721 not found: ID does not exist"
Oct 30 03:16:17.618652 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.618622    4334 scope.go:122] "RemoveContainer" containerID="0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f"
Oct 30 03:16:17.618879 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:17.618859    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f\": container with ID starting with 0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f not found: ID does not exist" containerID="0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f"
Oct 30 03:16:17.618921 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.618885    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f"} err="failed to get container status \"0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f\": rpc error: code = NotFound desc = could not find container \"0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f\": container with ID starting with 0ee8875db83d45aba64bf7b0cce9eab04e3337f58f879bac786d770f9e9c030f not found: ID does not exist"
Oct 30 03:16:17.654931 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.654911    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:16:17.666171 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:17.666151    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:16:19.158413 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:19.158373    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7498af25-0e73-49de-acad-64efeff7f900" path="/var/lib/kubelet/pods/7498af25-0e73-49de-acad-64efeff7f900/volumes"
Oct 30 03:16:20.977855 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:20.977822    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:16:21.910846 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:21.910804    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:21.911074 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:21.911020    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a60c3004-9c03-49ca-b1e7-4d7a92c983f6" containerName="otc-container" containerID="cri-o://bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e" gracePeriod=30
Oct 30 03:16:23.151857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.151831    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.290342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.290264    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otc-internal\") pod \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") "
Oct 30 03:16:23.290342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.290312    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-kube-api-access-hkg8d\" (UniqueName: \"kubernetes.io/projected/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-kube-api-access-hkg8d\") pod \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") "
Oct 30 03:16:23.290342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.290341    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-data-science-collector-ta-client-cert\") pod \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") "
Oct 30 03:16:23.290657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.290364    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otlp-tls-certs\") pod \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") "
Oct 30 03:16:23.290657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.290443    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-monitoring-tls-certs\") pod \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") "
Oct 30 03:16:23.290657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.290491    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-tls-certs\") pod \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\" (UID: \"a60c3004-9c03-49ca-b1e7-4d7a92c983f6\") "
Oct 30 03:16:23.290823 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.290736    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otc-internal" pod "a60c3004-9c03-49ca-b1e7-4d7a92c983f6" (UID: "a60c3004-9c03-49ca-b1e7-4d7a92c983f6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:23.292641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.292609    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otlp-tls-certs" pod "a60c3004-9c03-49ca-b1e7-4d7a92c983f6" (UID: "a60c3004-9c03-49ca-b1e7-4d7a92c983f6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:23.292641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.292624    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-kube-api-access-hkg8d" pod "a60c3004-9c03-49ca-b1e7-4d7a92c983f6" (UID: "a60c3004-9c03-49ca-b1e7-4d7a92c983f6"). InnerVolumeSpecName "kube-api-access-hkg8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:23.292641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.292633    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-monitoring-tls-certs" pod "a60c3004-9c03-49ca-b1e7-4d7a92c983f6" (UID: "a60c3004-9c03-49ca-b1e7-4d7a92c983f6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:23.292929 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.292913    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-tls-certs" pod "a60c3004-9c03-49ca-b1e7-4d7a92c983f6" (UID: "a60c3004-9c03-49ca-b1e7-4d7a92c983f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:23.293002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.292979    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-data-science-collector-ta-client-cert" pod "a60c3004-9c03-49ca-b1e7-4d7a92c983f6" (UID: "a60c3004-9c03-49ca-b1e7-4d7a92c983f6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:23.391962 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.391931    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkg8d\" (UniqueName: \"kubernetes.io/projected/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-kube-api-access-hkg8d\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:23.391962 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.391958    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:23.391962 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.391969    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:23.392152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.391977    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:23.392152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.391988    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:23.392152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.391996    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a60c3004-9c03-49ca-b1e7-4d7a92c983f6-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:23.626091 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.626001    4334 generic.go:356] "Generic (PLEG): container finished" podID="a60c3004-9c03-49ca-b1e7-4d7a92c983f6" containerID="bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e" exitCode=0
Oct 30 03:16:23.626091 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.626044    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.626307 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.626089    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a60c3004-9c03-49ca-b1e7-4d7a92c983f6","Type":"ContainerDied","Data":"bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e"}
Oct 30 03:16:23.626307 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.626144    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a60c3004-9c03-49ca-b1e7-4d7a92c983f6","Type":"ContainerDied","Data":"2ab0314841d4adebc1e4bca58fa4e0eec0988051b882f764c2b372a7385d6b27"}
Oct 30 03:16:23.626307 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.626166    4334 scope.go:122] "RemoveContainer" containerID="bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e"
Oct 30 03:16:23.635619 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.635604    4334 scope.go:122] "RemoveContainer" containerID="bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e"
Oct 30 03:16:23.635868 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:23.635850    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e\": container with ID starting with bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e not found: ID does not exist" containerID="bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e"
Oct 30 03:16:23.635934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.635875    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e"} err="failed to get container status \"bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e\": rpc error: code = NotFound desc = could not find container \"bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e\": container with ID starting with bc05955b9bded6f15b05b38bb6d2f4bb62d2da3cc6ba54d90ae3bb26d747206e not found: ID does not exist"
Oct 30 03:16:23.674048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.674009    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:23.685152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.685121    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:23.715956 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.715932    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:23.716271 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716259    4334 cpu_manager.go:414] "containerMap: removing container" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo-gateway-opa"
Oct 30 03:16:23.716329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716274    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo-gateway-opa"
Oct 30 03:16:23.716329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716297    4334 cpu_manager.go:414] "containerMap: removing container" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo-gateway"
Oct 30 03:16:23.716329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716302    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo-gateway"
Oct 30 03:16:23.716329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716317    4334 cpu_manager.go:414] "containerMap: removing container" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo"
Oct 30 03:16:23.716329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716322    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo"
Oct 30 03:16:23.716515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716334    4334 cpu_manager.go:414] "containerMap: removing container" podUID="a60c3004-9c03-49ca-b1e7-4d7a92c983f6" containerName="otc-container"
Oct 30 03:16:23.716515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716339    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60c3004-9c03-49ca-b1e7-4d7a92c983f6" containerName="otc-container"
Oct 30 03:16:23.716515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716389    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo-gateway"
Oct 30 03:16:23.716515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716398    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo"
Oct 30 03:16:23.716515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716404    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60c3004-9c03-49ca-b1e7-4d7a92c983f6" containerName="otc-container"
Oct 30 03:16:23.716515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.716410    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="7498af25-0e73-49de-acad-64efeff7f900" containerName="tempo-gateway-opa"
Oct 30 03:16:23.720824 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.720810    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.735203 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.735160    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:23.735203 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.735168    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:23.735364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.735316    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:23.735364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.735322    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:16:23.735437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.735366    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:23.737313 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.737290    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:23.745997 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.745978    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:23.897110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.897028    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.897110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.897068    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.897110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.897102    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.897385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.897162    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kl2f\" (UniqueName: \"kubernetes.io/projected/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-kube-api-access-7kl2f\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.897385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.897274    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.897385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.897318    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.997743 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.997717    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.997902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.997754    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.997902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.997778    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.997902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.997804    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.997902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.997854    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.997902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.997870    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl2f\" (UniqueName: \"kubernetes.io/projected/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-kube-api-access-7kl2f\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:23.998629 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:23.998590    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:24.000393 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.000368    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:24.000539 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.000517    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:24.000593 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.000574    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:24.000632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.000574    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:24.016333 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.016313    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kl2f\" (UniqueName: \"kubernetes.io/projected/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-kube-api-access-7kl2f\") pod \"data-science-collector-collector-1\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:24.034103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.034084    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:24.329691 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.329659    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:24.333697 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:16:24.333666    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6cd2a9_ad43_4b86_9908_a6ce77d476c9.slice/crio-62ae1e2e77f36f2d108fedb428e0b4cbb219564a7116b96d958e972477592d50 WatchSource:0}: Error finding container 62ae1e2e77f36f2d108fedb428e0b4cbb219564a7116b96d958e972477592d50: Status 404 returned error can't find the container with id 62ae1e2e77f36f2d108fedb428e0b4cbb219564a7116b96d958e972477592d50
Oct 30 03:16:24.335744 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.335709    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:24.335821 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.335769    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:24.633414 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.633333    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9","Type":"ContainerStarted","Data":"be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff"}
Oct 30 03:16:24.633414 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.633386    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9","Type":"ContainerStarted","Data":"62ae1e2e77f36f2d108fedb428e0b4cbb219564a7116b96d958e972477592d50"}
Oct 30 03:16:24.671976 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:24.671928    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6719134420000001 podStartE2EDuration="1.671913442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:24.670564468 +0000 UTC m=+2063.677154725" watchObservedRunningTime="2026-10-30 03:16:24.671913442 +0000 UTC m=+2063.678503696"
Oct 30 03:16:25.158876 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:25.158845    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60c3004-9c03-49ca-b1e7-4d7a92c983f6" path="/var/lib/kubelet/pods/a60c3004-9c03-49ca-b1e7-4d7a92c983f6/volumes"
Oct 30 03:16:30.689752 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.689714    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:30.690298 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.689945    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" containerName="otc-container" containerID="cri-o://be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff" gracePeriod=30
Oct 30 03:16:30.940753 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.940703    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:30.962018 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.961988    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-data-science-collector-ta-client-cert\") pod \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") "
Oct 30 03:16:30.962191 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.962048    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-kube-api-access-7kl2f\" (UniqueName: \"kubernetes.io/projected/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-kube-api-access-7kl2f\") pod \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") "
Oct 30 03:16:30.962191 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.962085    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-monitoring-tls-certs\") pod \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") "
Oct 30 03:16:30.962191 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.962161    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-tls-certs\") pod \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") "
Oct 30 03:16:30.962379 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.962208    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otlp-tls-certs\") pod \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") "
Oct 30 03:16:30.962379 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.962275    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otc-internal\") pod \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\" (UID: \"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9\") "
Oct 30 03:16:30.962940 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.962912    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otc-internal" pod "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" (UID: "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:30.965725 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.964557    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-tls-certs" pod "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" (UID: "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:30.965725 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.964602    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-monitoring-tls-certs" pod "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" (UID: "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:30.965725 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.964737    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otlp-tls-certs" pod "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" (UID: "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:30.965725 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.964745    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-kube-api-access-7kl2f" pod "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" (UID: "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9"). InnerVolumeSpecName "kube-api-access-7kl2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:30.965725 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:30.964988    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-data-science-collector-ta-client-cert" pod "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" (UID: "1e6cd2a9-ad43-4b86-9908-a6ce77d476c9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:31.063024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.062992    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:31.063024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.063016    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kl2f\" (UniqueName: \"kubernetes.io/projected/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-kube-api-access-7kl2f\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:31.063024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.063025    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:31.063269 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.063035    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:31.063269 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.063044    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:31.063269 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.063051    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:31.669566 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.669535    4334 generic.go:356] "Generic (PLEG): container finished" podID="1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" containerID="be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff" exitCode=0
Oct 30 03:16:31.669740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.669603    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.669740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.669625    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9","Type":"ContainerDied","Data":"be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff"}
Oct 30 03:16:31.669740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.669676    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1e6cd2a9-ad43-4b86-9908-a6ce77d476c9","Type":"ContainerDied","Data":"62ae1e2e77f36f2d108fedb428e0b4cbb219564a7116b96d958e972477592d50"}
Oct 30 03:16:31.669740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.669698    4334 scope.go:122] "RemoveContainer" containerID="be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff"
Oct 30 03:16:31.680209 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.680196    4334 scope.go:122] "RemoveContainer" containerID="be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff"
Oct 30 03:16:31.680559 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:31.680537    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff\": container with ID starting with be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff not found: ID does not exist" containerID="be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff"
Oct 30 03:16:31.680641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.680567    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff"} err="failed to get container status \"be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff\": rpc error: code = NotFound desc = could not find container \"be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff\": container with ID starting with be6b816c1ff9a808e13f36a942fc784ce3bcadd33668f37ab119ddc79e92e3ff not found: ID does not exist"
Oct 30 03:16:31.716304 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.716277    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:31.727728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.727710    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:31.755163 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.755141    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:31.755498 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.755472    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" containerName="otc-container"
Oct 30 03:16:31.755498 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.755485    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" containerName="otc-container"
Oct 30 03:16:31.755588 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.755554    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" containerName="otc-container"
Oct 30 03:16:31.759988 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.759968    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.775551 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.775534    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:31.775700 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.775587    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:31.775830 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.775604    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:31.775896 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.775635    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:31.775896 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.775722    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:31.775998 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.775952    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:31.784909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.784893    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:16:31.869955 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.869918    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.870077 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.869977    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.870077 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.870061    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wwpb\" (UniqueName: \"kubernetes.io/projected/11501c37-db80-40a0-ac4d-7bb504d10f60-kube-api-access-4wwpb\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.870151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.870134    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/11501c37-db80-40a0-ac4d-7bb504d10f60-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.870185 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.870164    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.870185 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.870180    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.970913 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.970836    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwpb\" (UniqueName: \"kubernetes.io/projected/11501c37-db80-40a0-ac4d-7bb504d10f60-kube-api-access-4wwpb\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.970913 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.970873    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/11501c37-db80-40a0-ac4d-7bb504d10f60-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.970913 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.970895    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.970913 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.970910    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.971167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.970932    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.971167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.970985    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.971636 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.971610    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/11501c37-db80-40a0-ac4d-7bb504d10f60-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.973541 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.973514    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.973752 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.973733    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.973823 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.973765    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.973823 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.973774    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:31.988746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:31.988727    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwpb\" (UniqueName: \"kubernetes.io/projected/11501c37-db80-40a0-ac4d-7bb504d10f60-kube-api-access-4wwpb\") pod \"data-science-collector-collector-1\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:32.072389 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:32.072368    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:32.366799 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:32.366716    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:32.370388 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:16:32.370357    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11501c37_db80_40a0_ac4d_7bb504d10f60.slice/crio-d08edd984d6cb38e7303bfa15ab4fe5cae7798f92ca47939d37cfbaae478de38 WatchSource:0}: Error finding container d08edd984d6cb38e7303bfa15ab4fe5cae7798f92ca47939d37cfbaae478de38: Status 404 returned error can't find the container with id d08edd984d6cb38e7303bfa15ab4fe5cae7798f92ca47939d37cfbaae478de38
Oct 30 03:16:32.372504 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:32.372471    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:32.372582 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:32.372529    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:32.676273 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:32.676239    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"11501c37-db80-40a0-ac4d-7bb504d10f60","Type":"ContainerStarted","Data":"fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c"}
Oct 30 03:16:32.676460 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:32.676292    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"11501c37-db80-40a0-ac4d-7bb504d10f60","Type":"ContainerStarted","Data":"d08edd984d6cb38e7303bfa15ab4fe5cae7798f92ca47939d37cfbaae478de38"}
Oct 30 03:16:32.716564 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:32.716514    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7164991870000001 podStartE2EDuration="1.716499187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:32.714815683 +0000 UTC m=+2071.721405939" watchObservedRunningTime="2026-10-30 03:16:32.716499187 +0000 UTC m=+2071.723089500"
Oct 30 03:16:33.159206 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:33.159181    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e6cd2a9-ad43-4b86-9908-a6ce77d476c9" path="/var/lib/kubelet/pods/1e6cd2a9-ad43-4b86-9908-a6ce77d476c9/volumes"
Oct 30 03:16:44.238647 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:44.238613    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:44.239233 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:44.238858    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="11501c37-db80-40a0-ac4d-7bb504d10f60" containerName="otc-container" containerID="cri-o://fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c" gracePeriod=30
Oct 30 03:16:46.575438 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.575414    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.690477 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.690434    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-tls-certs\") pod \"11501c37-db80-40a0-ac4d-7bb504d10f60\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") "
Oct 30 03:16:46.690649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.690503    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11501c37-db80-40a0-ac4d-7bb504d10f60-otc-internal\" (UniqueName: \"kubernetes.io/configmap/11501c37-db80-40a0-ac4d-7bb504d10f60-otc-internal\") pod \"11501c37-db80-40a0-ac4d-7bb504d10f60\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") "
Oct 30 03:16:46.690649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.690530    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-monitoring-tls-certs\") pod \"11501c37-db80-40a0-ac4d-7bb504d10f60\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") "
Oct 30 03:16:46.690649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.690549    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-data-science-collector-ta-client-cert\") pod \"11501c37-db80-40a0-ac4d-7bb504d10f60\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") "
Oct 30 03:16:46.690649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.690581    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11501c37-db80-40a0-ac4d-7bb504d10f60-kube-api-access-4wwpb\" (UniqueName: \"kubernetes.io/projected/11501c37-db80-40a0-ac4d-7bb504d10f60-kube-api-access-4wwpb\") pod \"11501c37-db80-40a0-ac4d-7bb504d10f60\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") "
Oct 30 03:16:46.690649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.690607    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-otlp-tls-certs\") pod \"11501c37-db80-40a0-ac4d-7bb504d10f60\" (UID: \"11501c37-db80-40a0-ac4d-7bb504d10f60\") "
Oct 30 03:16:46.690990 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.690959    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11501c37-db80-40a0-ac4d-7bb504d10f60-otc-internal" pod "11501c37-db80-40a0-ac4d-7bb504d10f60" (UID: "11501c37-db80-40a0-ac4d-7bb504d10f60"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:16:46.692924 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.692896    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-monitoring-tls-certs" pod "11501c37-db80-40a0-ac4d-7bb504d10f60" (UID: "11501c37-db80-40a0-ac4d-7bb504d10f60"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:46.692924 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.692894    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11501c37-db80-40a0-ac4d-7bb504d10f60-kube-api-access-4wwpb" pod "11501c37-db80-40a0-ac4d-7bb504d10f60" (UID: "11501c37-db80-40a0-ac4d-7bb504d10f60"). InnerVolumeSpecName "kube-api-access-4wwpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:16:46.692924 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.692912    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-otlp-tls-certs" pod "11501c37-db80-40a0-ac4d-7bb504d10f60" (UID: "11501c37-db80-40a0-ac4d-7bb504d10f60"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:46.693109 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.693096    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-data-science-collector-ta-client-cert" pod "11501c37-db80-40a0-ac4d-7bb504d10f60" (UID: "11501c37-db80-40a0-ac4d-7bb504d10f60"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:46.693162 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.693119    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-tls-certs" pod "11501c37-db80-40a0-ac4d-7bb504d10f60" (UID: "11501c37-db80-40a0-ac4d-7bb504d10f60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:16:46.747777 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.747749    4334 generic.go:356] "Generic (PLEG): container finished" podID="11501c37-db80-40a0-ac4d-7bb504d10f60" containerID="fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c" exitCode=0
Oct 30 03:16:46.747886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.747790    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"11501c37-db80-40a0-ac4d-7bb504d10f60","Type":"ContainerDied","Data":"fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c"}
Oct 30 03:16:46.747886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.747808    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.747886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.747837    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"11501c37-db80-40a0-ac4d-7bb504d10f60","Type":"ContainerDied","Data":"d08edd984d6cb38e7303bfa15ab4fe5cae7798f92ca47939d37cfbaae478de38"}
Oct 30 03:16:46.747886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.747860    4334 scope.go:122] "RemoveContainer" containerID="fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c"
Oct 30 03:16:46.756798 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.756724    4334 scope.go:122] "RemoveContainer" containerID="fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c"
Oct 30 03:16:46.757145 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:46.757121    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c\": container with ID starting with fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c not found: ID does not exist" containerID="fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c"
Oct 30 03:16:46.757241 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.757156    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c"} err="failed to get container status \"fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c\": rpc error: code = NotFound desc = could not find container \"fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c\": container with ID starting with fc6240b9ef7c332578992a26493fa9e916effcdb3ebd14569ecce4eaf30c8e5c not found: ID does not exist"
Oct 30 03:16:46.792272 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.792254    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/11501c37-db80-40a0-ac4d-7bb504d10f60-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:46.792369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.792277    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:46.792369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.792286    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:46.792369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.792295    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wwpb\" (UniqueName: \"kubernetes.io/projected/11501c37-db80-40a0-ac4d-7bb504d10f60-kube-api-access-4wwpb\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:46.792369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.792304    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:46.792369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.792312    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11501c37-db80-40a0-ac4d-7bb504d10f60-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:16:46.794571 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.794553    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:46.798793 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.798772    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:46.836132 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.836108    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:46.836527 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.836511    4334 cpu_manager.go:414] "containerMap: removing container" podUID="11501c37-db80-40a0-ac4d-7bb504d10f60" containerName="otc-container"
Oct 30 03:16:46.836527 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.836526    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11501c37-db80-40a0-ac4d-7bb504d10f60" containerName="otc-container"
Oct 30 03:16:46.836663 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.836587    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="11501c37-db80-40a0-ac4d-7bb504d10f60" containerName="otc-container"
Oct 30 03:16:46.841056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.841041    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.856134 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.856109    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:16:46.859242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.859217    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:16:46.859339 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.859248    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:16:46.859608 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.859586    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:16:46.859750 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.859733    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:16:46.860073 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.859990    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:46.861774 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.860639    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:16:46.892912 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.892882    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.893041 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.892935    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.893041 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.893023    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/04a00494-320d-4d38-a409-a791e2d66230-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.893163 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.893071    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.893163 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.893134    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.893251 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.893197    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5wl\" (UniqueName: \"kubernetes.io/projected/04a00494-320d-4d38-a409-a791e2d66230-kube-api-access-sq5wl\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.994345 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.994280    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.994345 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.994338    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.994564 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.994367    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/04a00494-320d-4d38-a409-a791e2d66230-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.994564 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.994487    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.994564 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.994526    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.994718 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.994565    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5wl\" (UniqueName: \"kubernetes.io/projected/04a00494-320d-4d38-a409-a791e2d66230-kube-api-access-sq5wl\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.995104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.995082    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/04a00494-320d-4d38-a409-a791e2d66230-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.996865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.996836    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.996965 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.996937    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.997222 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.997204    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:46.997259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:46.997230    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:47.012366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.012348    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5wl\" (UniqueName: \"kubernetes.io/projected/04a00494-320d-4d38-a409-a791e2d66230-kube-api-access-sq5wl\") pod \"data-science-collector-collector-1\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:47.152740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.152712    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:16:47.158884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.158862    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11501c37-db80-40a0-ac4d-7bb504d10f60" path="/var/lib/kubelet/pods/11501c37-db80-40a0-ac4d-7bb504d10f60/volumes"
Oct 30 03:16:47.447945 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.447916    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:16:47.454674 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:16:47.454641    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a00494_320d_4d38_a409_a791e2d66230.slice/crio-5bc134cadf9fd9eba4f3b10c2179873d08863764bc928bc05b16516bb7a9e507 WatchSource:0}: Error finding container 5bc134cadf9fd9eba4f3b10c2179873d08863764bc928bc05b16516bb7a9e507: Status 404 returned error can't find the container with id 5bc134cadf9fd9eba4f3b10c2179873d08863764bc928bc05b16516bb7a9e507
Oct 30 03:16:47.456968 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.456932    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:47.457091 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.457003    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:16:47.754979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.754879    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"04a00494-320d-4d38-a409-a791e2d66230","Type":"ContainerStarted","Data":"56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2"}
Oct 30 03:16:47.754979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.754947    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"04a00494-320d-4d38-a409-a791e2d66230","Type":"ContainerStarted","Data":"5bc134cadf9fd9eba4f3b10c2179873d08863764bc928bc05b16516bb7a9e507"}
Oct 30 03:16:47.808820 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:16:47.808776    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.808763455 podStartE2EDuration="1.808763455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:47.796890538 +0000 UTC m=+2086.803480792" watchObservedRunningTime="2026-10-30 03:16:47.808763455 +0000 UTC m=+2086.815353709"
Oct 30 03:16:50.978245 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:16:50.978209    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:17:20.978581 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:17:20.978488    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:17:50.978798 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:17:50.978762    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:18:20.979233 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:18:20.979199    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:18:50.978133 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:18:50.978040    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:19:20.977861 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:19:20.977825    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:19:50.978372 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:19:50.978335    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:20:20.977833 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:20:20.977748    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:20:50.978009 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:20:50.977972    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:21:20.978842 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:21:20.978804    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:21:50.978200 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:21:50.978111    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:22:20.979096 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:22:20.978990    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:22:50.978154 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:22:50.978119    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:23:20.978643 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:23:20.978567    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:23:50.978294 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:23:50.978244    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:24:20.977638 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:24:20.977591    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:24:50.978469 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:24:50.978355    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:25:20.977931 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:25:20.977887    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:25:50.978244 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:25:50.978205    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:26:20.978297 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:26:20.978217    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:26:46.494652 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:46.494616    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:26:46.495320 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:46.494885    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="04a00494-320d-4d38-a409-a791e2d66230" containerName="otc-container" containerID="cri-o://56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2" gracePeriod=30
Oct 30 03:26:48.837063 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.837031    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:48.939777 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.939747    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a00494-320d-4d38-a409-a791e2d66230-kube-api-access-sq5wl\" (UniqueName: \"kubernetes.io/projected/04a00494-320d-4d38-a409-a791e2d66230-kube-api-access-sq5wl\") pod \"04a00494-320d-4d38-a409-a791e2d66230\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") "
Oct 30 03:26:48.939939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.939794    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a00494-320d-4d38-a409-a791e2d66230-otc-internal\" (UniqueName: \"kubernetes.io/configmap/04a00494-320d-4d38-a409-a791e2d66230-otc-internal\") pod \"04a00494-320d-4d38-a409-a791e2d66230\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") "
Oct 30 03:26:48.939939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.939826    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-tls-certs\") pod \"04a00494-320d-4d38-a409-a791e2d66230\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") "
Oct 30 03:26:48.939939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.939854    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-monitoring-tls-certs\") pod \"04a00494-320d-4d38-a409-a791e2d66230\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") "
Oct 30 03:26:48.939939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.939897    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-data-science-collector-ta-client-cert\") pod \"04a00494-320d-4d38-a409-a791e2d66230\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") "
Oct 30 03:26:48.940165 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.939940    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-otlp-tls-certs\") pod \"04a00494-320d-4d38-a409-a791e2d66230\" (UID: \"04a00494-320d-4d38-a409-a791e2d66230\") "
Oct 30 03:26:48.940228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.940191    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a00494-320d-4d38-a409-a791e2d66230-otc-internal" pod "04a00494-320d-4d38-a409-a791e2d66230" (UID: "04a00494-320d-4d38-a409-a791e2d66230"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:48.940346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.940329    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/04a00494-320d-4d38-a409-a791e2d66230-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:48.942038 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.942005    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-monitoring-tls-certs" pod "04a00494-320d-4d38-a409-a791e2d66230" (UID: "04a00494-320d-4d38-a409-a791e2d66230"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:48.942163 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.942121    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a00494-320d-4d38-a409-a791e2d66230-kube-api-access-sq5wl" pod "04a00494-320d-4d38-a409-a791e2d66230" (UID: "04a00494-320d-4d38-a409-a791e2d66230"). InnerVolumeSpecName "kube-api-access-sq5wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:48.942256 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.942163    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-data-science-collector-ta-client-cert" pod "04a00494-320d-4d38-a409-a791e2d66230" (UID: "04a00494-320d-4d38-a409-a791e2d66230"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:48.942502 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.942478    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-otlp-tls-certs" pod "04a00494-320d-4d38-a409-a791e2d66230" (UID: "04a00494-320d-4d38-a409-a791e2d66230"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:48.942502 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:48.942493    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-tls-certs" pod "04a00494-320d-4d38-a409-a791e2d66230" (UID: "04a00494-320d-4d38-a409-a791e2d66230"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:49.041235 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.041208    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:49.041235 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.041233    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:49.041235 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.041243    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:49.041436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.041252    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq5wl\" (UniqueName: \"kubernetes.io/projected/04a00494-320d-4d38-a409-a791e2d66230-kube-api-access-sq5wl\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:49.041436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.041262    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a00494-320d-4d38-a409-a791e2d66230-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:49.608904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.608872    4334 generic.go:356] "Generic (PLEG): container finished" podID="04a00494-320d-4d38-a409-a791e2d66230" containerID="56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2" exitCode=0
Oct 30 03:26:49.609098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.608925    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.609098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.608961    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"04a00494-320d-4d38-a409-a791e2d66230","Type":"ContainerDied","Data":"56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2"}
Oct 30 03:26:49.609098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.609022    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"04a00494-320d-4d38-a409-a791e2d66230","Type":"ContainerDied","Data":"5bc134cadf9fd9eba4f3b10c2179873d08863764bc928bc05b16516bb7a9e507"}
Oct 30 03:26:49.609098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.609053    4334 scope.go:122] "RemoveContainer" containerID="56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2"
Oct 30 03:26:49.617837 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.617823    4334 scope.go:122] "RemoveContainer" containerID="56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2"
Oct 30 03:26:49.618091 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:26:49.618072    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2\": container with ID starting with 56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2 not found: ID does not exist" containerID="56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2"
Oct 30 03:26:49.618135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.618101    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2"} err="failed to get container status \"56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2\": rpc error: code = NotFound desc = could not find container \"56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2\": container with ID starting with 56bd845fe2bc4d61ced98a04625c7da70f697d847f297cb2d53f09c79f301cc2 not found: ID does not exist"
Oct 30 03:26:49.657175 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.657151    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:26:49.667885 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.667854    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:26:49.695508 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.695485    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:26:49.695818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.695807    4334 cpu_manager.go:414] "containerMap: removing container" podUID="04a00494-320d-4d38-a409-a791e2d66230" containerName="otc-container"
Oct 30 03:26:49.695860 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.695820    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a00494-320d-4d38-a409-a791e2d66230" containerName="otc-container"
Oct 30 03:26:49.695977 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.695963    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a00494-320d-4d38-a409-a791e2d66230" containerName="otc-container"
Oct 30 03:26:49.700635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.700601    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.718961 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.718930    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:26:49.719375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.719098    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:26:49.719375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.719189    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:26:49.719375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.719251    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:26:49.719375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.719326    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:26:49.720003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.719981    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:26:49.728364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.728348    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:26:49.849437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.849405    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.849801 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.849442    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.849801 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.849552    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hg54\" (UniqueName: \"kubernetes.io/projected/b9438044-2ff2-4f64-a59a-5c5cddaddf76-kube-api-access-9hg54\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.849801 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.849595    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.849801 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.849654    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.849801 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.849676    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.950614 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.950584    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.950614 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.950621    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.950865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.950640    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.950865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.950713    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.950865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.950747    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.950865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.950779    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hg54\" (UniqueName: \"kubernetes.io/projected/b9438044-2ff2-4f64-a59a-5c5cddaddf76-kube-api-access-9hg54\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.951654 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.951627    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.953904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.953881    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.954043 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.953913    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.954205 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.954185    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.954747 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.954725    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:49.970935 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:49.970914    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hg54\" (UniqueName: \"kubernetes.io/projected/b9438044-2ff2-4f64-a59a-5c5cddaddf76-kube-api-access-9hg54\") pod \"data-science-collector-collector-1\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:50.013331 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:50.013301    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:50.312457 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:50.312369    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:26:50.317817 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:26:50.317790    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9438044_2ff2_4f64_a59a_5c5cddaddf76.slice/crio-4f017c85034022b4570a537637df56c69d9137417ee350e2d40338a86421decc WatchSource:0}: Error finding container 4f017c85034022b4570a537637df56c69d9137417ee350e2d40338a86421decc: Status 404 returned error can't find the container with id 4f017c85034022b4570a537637df56c69d9137417ee350e2d40338a86421decc
Oct 30 03:26:50.319883 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:50.319849    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:26:50.319982 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:50.319912    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:26:50.616857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:50.616736    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b9438044-2ff2-4f64-a59a-5c5cddaddf76","Type":"ContainerStarted","Data":"f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c"}
Oct 30 03:26:50.616857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:50.616820    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b9438044-2ff2-4f64-a59a-5c5cddaddf76","Type":"ContainerStarted","Data":"4f017c85034022b4570a537637df56c69d9137417ee350e2d40338a86421decc"}
Oct 30 03:26:50.655561 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:50.655514    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6554980000000001 podStartE2EDuration="1.655498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:50.654770631 +0000 UTC m=+2689.661360885" watchObservedRunningTime="2026-10-30 03:26:50.655498 +0000 UTC m=+2689.662088256"
Oct 30 03:26:50.980463 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:26:50.980424    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:26:51.162895 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:51.162856    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a00494-320d-4d38-a409-a791e2d66230" path="/var/lib/kubelet/pods/04a00494-320d-4d38-a409-a791e2d66230/volumes"
Oct 30 03:26:58.546802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:58.546768    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:26:58.547264 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:58.546971    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b9438044-2ff2-4f64-a59a-5c5cddaddf76" containerName="otc-container" containerID="cri-o://f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c" gracePeriod=30
Oct 30 03:26:59.791229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.791209    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:26:59.823518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.823487    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otlp-tls-certs\") pod \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") "
Oct 30 03:26:59.823623 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.823555    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-monitoring-tls-certs\") pod \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") "
Oct 30 03:26:59.823671 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.823653    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-tls-certs\") pod \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") "
Oct 30 03:26:59.823718 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.823700    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otc-internal\") pod \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") "
Oct 30 03:26:59.823793 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.823773    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-data-science-collector-ta-client-cert\") pod \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") "
Oct 30 03:26:59.823906 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.823825    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9438044-2ff2-4f64-a59a-5c5cddaddf76-kube-api-access-9hg54\" (UniqueName: \"kubernetes.io/projected/b9438044-2ff2-4f64-a59a-5c5cddaddf76-kube-api-access-9hg54\") pod \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\" (UID: \"b9438044-2ff2-4f64-a59a-5c5cddaddf76\") "
Oct 30 03:26:59.824117 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.824080    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otc-internal" pod "b9438044-2ff2-4f64-a59a-5c5cddaddf76" (UID: "b9438044-2ff2-4f64-a59a-5c5cddaddf76"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:59.825836 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.825807    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-tls-certs" pod "b9438044-2ff2-4f64-a59a-5c5cddaddf76" (UID: "b9438044-2ff2-4f64-a59a-5c5cddaddf76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:59.825949 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.825872    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-data-science-collector-ta-client-cert" pod "b9438044-2ff2-4f64-a59a-5c5cddaddf76" (UID: "b9438044-2ff2-4f64-a59a-5c5cddaddf76"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:59.825949 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.825917    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9438044-2ff2-4f64-a59a-5c5cddaddf76-kube-api-access-9hg54" pod "b9438044-2ff2-4f64-a59a-5c5cddaddf76" (UID: "b9438044-2ff2-4f64-a59a-5c5cddaddf76"). InnerVolumeSpecName "kube-api-access-9hg54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:59.826120 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.826101    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-monitoring-tls-certs" pod "b9438044-2ff2-4f64-a59a-5c5cddaddf76" (UID: "b9438044-2ff2-4f64-a59a-5c5cddaddf76"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:59.826317 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.826301    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otlp-tls-certs" pod "b9438044-2ff2-4f64-a59a-5c5cddaddf76" (UID: "b9438044-2ff2-4f64-a59a-5c5cddaddf76"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:26:59.926288 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.926254    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hg54\" (UniqueName: \"kubernetes.io/projected/b9438044-2ff2-4f64-a59a-5c5cddaddf76-kube-api-access-9hg54\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:59.926288 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.926287    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:59.926561 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.926300    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:59.926561 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.926314    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:59.926561 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.926329    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9438044-2ff2-4f64-a59a-5c5cddaddf76-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:26:59.926561 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:26:59.926343    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9438044-2ff2-4f64-a59a-5c5cddaddf76-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:27:00.232388 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.232318    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"]
Oct 30 03:27:00.232901 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.232883    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b9438044-2ff2-4f64-a59a-5c5cddaddf76" containerName="otc-container"
Oct 30 03:27:00.232961 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.232905    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9438044-2ff2-4f64-a59a-5c5cddaddf76" containerName="otc-container"
Oct 30 03:27:00.233025 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.233014    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9438044-2ff2-4f64-a59a-5c5cddaddf76" containerName="otc-container"
Oct 30 03:27:00.236731 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.236711    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:27:00.256396 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.256371    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"]
Oct 30 03:27:00.257706 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.257670    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"kube-root-ca.crt\""
Oct 30 03:27:00.257840 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.257722    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"openshift-service-ca.crt\""
Oct 30 03:27:00.298201 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.298179    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-tenant-metrics\"/\"default-dockercfg-ljrp4\""
Oct 30 03:27:00.329706 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.329671    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60e290ab-2016-4748-811e-7cc28c0e28af-data\") pod \"tenant-metrics-exporter-5cf544fb97-q7fjr\" (UID: \"60e290ab-2016-4748-811e-7cc28c0e28af\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:27:00.329845 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.329742    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqls\" (UniqueName: \"kubernetes.io/projected/60e290ab-2016-4748-811e-7cc28c0e28af-kube-api-access-xnqls\") pod \"tenant-metrics-exporter-5cf544fb97-q7fjr\" (UID: \"60e290ab-2016-4748-811e-7cc28c0e28af\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:27:00.430974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.430947    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqls\" (UniqueName: \"kubernetes.io/projected/60e290ab-2016-4748-811e-7cc28c0e28af-kube-api-access-xnqls\") pod \"tenant-metrics-exporter-5cf544fb97-q7fjr\" (UID: \"60e290ab-2016-4748-811e-7cc28c0e28af\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:27:00.431097 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.431002    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60e290ab-2016-4748-811e-7cc28c0e28af-data\") pod \"tenant-metrics-exporter-5cf544fb97-q7fjr\" (UID: \"60e290ab-2016-4748-811e-7cc28c0e28af\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:27:00.431345 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.431326    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60e290ab-2016-4748-811e-7cc28c0e28af-data\") pod \"tenant-metrics-exporter-5cf544fb97-q7fjr\" (UID: \"60e290ab-2016-4748-811e-7cc28c0e28af\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:27:00.452266 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.452244    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqls\" (UniqueName: \"kubernetes.io/projected/60e290ab-2016-4748-811e-7cc28c0e28af-kube-api-access-xnqls\") pod \"tenant-metrics-exporter-5cf544fb97-q7fjr\" (UID: \"60e290ab-2016-4748-811e-7cc28c0e28af\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:27:00.550586 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.550522    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:27:00.666053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.666023    4334 generic.go:356] "Generic (PLEG): container finished" podID="b9438044-2ff2-4f64-a59a-5c5cddaddf76" containerID="f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c" exitCode=0
Oct 30 03:27:00.666189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.666069    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.666189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.666068    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b9438044-2ff2-4f64-a59a-5c5cddaddf76","Type":"ContainerDied","Data":"f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c"}
Oct 30 03:27:00.666260 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.666200    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b9438044-2ff2-4f64-a59a-5c5cddaddf76","Type":"ContainerDied","Data":"4f017c85034022b4570a537637df56c69d9137417ee350e2d40338a86421decc"}
Oct 30 03:27:00.666260 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.666233    4334 scope.go:122] "RemoveContainer" containerID="f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c"
Oct 30 03:27:00.675460 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.675429    4334 scope.go:122] "RemoveContainer" containerID="f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c"
Oct 30 03:27:00.675752 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:27:00.675729    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c\": container with ID starting with f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c not found: ID does not exist" containerID="f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c"
Oct 30 03:27:00.675827 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.675763    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c"} err="failed to get container status \"f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c\": rpc error: code = NotFound desc = could not find container \"f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c\": container with ID starting with f5936728549ad4647346acbf1c87780af2a2d72ddca0ac248534aa43bd04998c not found: ID does not exist"
Oct 30 03:27:00.717023 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.717002    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:27:00.727429 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.727397    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:27:00.773907 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.773881    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:27:00.778728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.778712    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.799556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.799540    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:27:00.799888 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.799688    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:27:00.799888 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.799705    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:27:00.799888 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.799718    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:27:00.799888 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.799733    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:27:00.807234 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.807178    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:27:00.815778 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.815761    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:27:00.834881 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.834859    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8274001c-2f72-4782-af5f-cd16fb20bbde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.834967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.834890    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlqsk\" (UniqueName: \"kubernetes.io/projected/8274001c-2f72-4782-af5f-cd16fb20bbde-kube-api-access-hlqsk\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.834967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.834915    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.835039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.835020    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.835070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.835055    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.835101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.835079    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.862129 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.862109    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"]
Oct 30 03:27:00.878883 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:27:00.878851    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e290ab_2016_4748_811e_7cc28c0e28af.slice/crio-e32fce384333fac1708fab675cd10ae185b76d346bbcb555cb779c7083d0848d WatchSource:0}: Error finding container e32fce384333fac1708fab675cd10ae185b76d346bbcb555cb779c7083d0848d: Status 404 returned error can't find the container with id e32fce384333fac1708fab675cd10ae185b76d346bbcb555cb779c7083d0848d
Oct 30 03:27:00.880792 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.880775    4334 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:27:00.936129 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.936108    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8274001c-2f72-4782-af5f-cd16fb20bbde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.936230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.936139    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlqsk\" (UniqueName: \"kubernetes.io/projected/8274001c-2f72-4782-af5f-cd16fb20bbde-kube-api-access-hlqsk\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.936230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.936171    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.936230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.936204    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.936401 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.936233    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.936401 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.936265    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.936960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.936931    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8274001c-2f72-4782-af5f-cd16fb20bbde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.938718 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.938691    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.938799 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.938782    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.938865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.938850    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.938973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.938952    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:00.958036 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:00.958017    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlqsk\" (UniqueName: \"kubernetes.io/projected/8274001c-2f72-4782-af5f-cd16fb20bbde-kube-api-access-hlqsk\") pod \"data-science-collector-collector-1\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:01.090826 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.090747    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:27:01.159052 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.159022    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9438044-2ff2-4f64-a59a-5c5cddaddf76" path="/var/lib/kubelet/pods/b9438044-2ff2-4f64-a59a-5c5cddaddf76/volumes"
Oct 30 03:27:01.394285 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.394207    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:27:01.398397 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:27:01.398364    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8274001c_2f72_4782_af5f_cd16fb20bbde.slice/crio-b652276ab63b6aa6208e105d1320734152c379926f6060803ba330cfe449f471 WatchSource:0}: Error finding container b652276ab63b6aa6208e105d1320734152c379926f6060803ba330cfe449f471: Status 404 returned error can't find the container with id b652276ab63b6aa6208e105d1320734152c379926f6060803ba330cfe449f471
Oct 30 03:27:01.400485 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.400429    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:27:01.400567 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.400514    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:27:01.675425 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.675394    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr" event={"ID":"60e290ab-2016-4748-811e-7cc28c0e28af","Type":"ContainerStarted","Data":"e32fce384333fac1708fab675cd10ae185b76d346bbcb555cb779c7083d0848d"}
Oct 30 03:27:01.677226 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.677195    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8274001c-2f72-4782-af5f-cd16fb20bbde","Type":"ContainerStarted","Data":"148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3"}
Oct 30 03:27:01.677374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.677254    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8274001c-2f72-4782-af5f-cd16fb20bbde","Type":"ContainerStarted","Data":"b652276ab63b6aa6208e105d1320734152c379926f6060803ba330cfe449f471"}
Oct 30 03:27:01.717766 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:01.717680    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.717661894 podStartE2EDuration="1.717661894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:01.716019241 +0000 UTC m=+2700.722609495" watchObservedRunningTime="2026-10-30 03:27:01.717661894 +0000 UTC m=+2700.724252156"
Oct 30 03:27:04.695575 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:04.695541    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr" event={"ID":"60e290ab-2016-4748-811e-7cc28c0e28af","Type":"ContainerStarted","Data":"7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f"}
Oct 30 03:27:04.731915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:27:04.731866    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr" podStartSLOduration=1.07878923 podStartE2EDuration="4.731853994s" totalImagesPullingTime="3.653064764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:04.730605173 +0000 UTC m=+2703.737195429" watchObservedRunningTime="2026-10-30 03:27:04.731853994 +0000 UTC m=+2703.738444247"
Oct 30 03:27:20.978770 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:27:20.978726    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:27:50.978885 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:27:50.978796    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:28:20.979030 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:28:20.978993    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:28:37.236479 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.236411    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"]
Oct 30 03:28:37.237102 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.236598    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr" podUID="60e290ab-2016-4748-811e-7cc28c0e28af" containerName="tenant-metrics-exporter" containerID="cri-o://7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f" gracePeriod=30
Oct 30 03:28:37.479905 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.479877    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:28:37.501308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.501247    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e290ab-2016-4748-811e-7cc28c0e28af-data\" (UniqueName: \"kubernetes.io/empty-dir/60e290ab-2016-4748-811e-7cc28c0e28af-data\") pod \"60e290ab-2016-4748-811e-7cc28c0e28af\" (UID: \"60e290ab-2016-4748-811e-7cc28c0e28af\") "
Oct 30 03:28:37.501308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.501278    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e290ab-2016-4748-811e-7cc28c0e28af-kube-api-access-xnqls\" (UniqueName: \"kubernetes.io/projected/60e290ab-2016-4748-811e-7cc28c0e28af-kube-api-access-xnqls\") pod \"60e290ab-2016-4748-811e-7cc28c0e28af\" (UID: \"60e290ab-2016-4748-811e-7cc28c0e28af\") "
Oct 30 03:28:37.501919 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.501844    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e290ab-2016-4748-811e-7cc28c0e28af-data" pod "60e290ab-2016-4748-811e-7cc28c0e28af" (UID: "60e290ab-2016-4748-811e-7cc28c0e28af"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:37.504886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.504609    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e290ab-2016-4748-811e-7cc28c0e28af-kube-api-access-xnqls" pod "60e290ab-2016-4748-811e-7cc28c0e28af" (UID: "60e290ab-2016-4748-811e-7cc28c0e28af"). InnerVolumeSpecName "kube-api-access-xnqls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:37.601971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.601942    4334 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60e290ab-2016-4748-811e-7cc28c0e28af-data\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:28:37.601971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:37.601970    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqls\" (UniqueName: \"kubernetes.io/projected/60e290ab-2016-4748-811e-7cc28c0e28af-kube-api-access-xnqls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:28:38.141100 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.141067    4334 generic.go:356] "Generic (PLEG): container finished" podID="60e290ab-2016-4748-811e-7cc28c0e28af" containerID="7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f" exitCode=0
Oct 30 03:28:38.141297 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.141113    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr" event={"ID":"60e290ab-2016-4748-811e-7cc28c0e28af","Type":"ContainerDied","Data":"7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f"}
Oct 30 03:28:38.141297 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.141132    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"
Oct 30 03:28:38.141297 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.141177    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr" event={"ID":"60e290ab-2016-4748-811e-7cc28c0e28af","Type":"ContainerDied","Data":"e32fce384333fac1708fab675cd10ae185b76d346bbcb555cb779c7083d0848d"}
Oct 30 03:28:38.141297 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.141204    4334 scope.go:122] "RemoveContainer" containerID="7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f"
Oct 30 03:28:38.150202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.150185    4334 scope.go:122] "RemoveContainer" containerID="7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f"
Oct 30 03:28:38.150486 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:28:38.150437    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f\": container with ID starting with 7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f not found: ID does not exist" containerID="7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f"
Oct 30 03:28:38.150558 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.150499    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f"} err="failed to get container status \"7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f\": rpc error: code = NotFound desc = could not find container \"7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f\": container with ID starting with 7ba08f9205389dbc5d7cd4ae388398b80a32b6503e5d6a1ef46198de70d7882f not found: ID does not exist"
Oct 30 03:28:38.187973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.187951    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"]
Oct 30 03:28:38.199034 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:38.199013    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-q7fjr"]
Oct 30 03:28:39.159033 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:28:39.159002    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e290ab-2016-4748-811e-7cc28c0e28af" path="/var/lib/kubelet/pods/60e290ab-2016-4748-811e-7cc28c0e28af/volumes"
Oct 30 03:28:50.978058 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:28:50.978015    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:29:06.331053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:06.331019    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:06.331592 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:06.331230    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="8274001c-2f72-4782-af5f-cd16fb20bbde" containerName="otc-container" containerID="cri-o://148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3" gracePeriod=30
Oct 30 03:29:08.663740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.663717    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:08.759413 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.759338    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-otlp-tls-certs\") pod \"8274001c-2f72-4782-af5f-cd16fb20bbde\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") "
Oct 30 03:29:08.759413 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.759379    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8274001c-2f72-4782-af5f-cd16fb20bbde-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8274001c-2f72-4782-af5f-cd16fb20bbde-otc-internal\") pod \"8274001c-2f72-4782-af5f-cd16fb20bbde\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") "
Oct 30 03:29:08.759641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.759427    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-data-science-collector-ta-client-cert\") pod \"8274001c-2f72-4782-af5f-cd16fb20bbde\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") "
Oct 30 03:29:08.759641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.759510    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-monitoring-tls-certs\") pod \"8274001c-2f72-4782-af5f-cd16fb20bbde\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") "
Oct 30 03:29:08.759641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.759558    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-tls-certs\") pod \"8274001c-2f72-4782-af5f-cd16fb20bbde\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") "
Oct 30 03:29:08.759641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.759620    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8274001c-2f72-4782-af5f-cd16fb20bbde-kube-api-access-hlqsk\" (UniqueName: \"kubernetes.io/projected/8274001c-2f72-4782-af5f-cd16fb20bbde-kube-api-access-hlqsk\") pod \"8274001c-2f72-4782-af5f-cd16fb20bbde\" (UID: \"8274001c-2f72-4782-af5f-cd16fb20bbde\") "
Oct 30 03:29:08.760147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.760117    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8274001c-2f72-4782-af5f-cd16fb20bbde-otc-internal" pod "8274001c-2f72-4782-af5f-cd16fb20bbde" (UID: "8274001c-2f72-4782-af5f-cd16fb20bbde"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:08.761834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.761797    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-tls-certs" pod "8274001c-2f72-4782-af5f-cd16fb20bbde" (UID: "8274001c-2f72-4782-af5f-cd16fb20bbde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:08.762192 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.762171    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-data-science-collector-ta-client-cert" pod "8274001c-2f72-4782-af5f-cd16fb20bbde" (UID: "8274001c-2f72-4782-af5f-cd16fb20bbde"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:08.762266 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.762194    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-otlp-tls-certs" pod "8274001c-2f72-4782-af5f-cd16fb20bbde" (UID: "8274001c-2f72-4782-af5f-cd16fb20bbde"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:08.762266 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.762208    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-monitoring-tls-certs" pod "8274001c-2f72-4782-af5f-cd16fb20bbde" (UID: "8274001c-2f72-4782-af5f-cd16fb20bbde"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:08.762378 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.762282    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8274001c-2f72-4782-af5f-cd16fb20bbde-kube-api-access-hlqsk" pod "8274001c-2f72-4782-af5f-cd16fb20bbde" (UID: "8274001c-2f72-4782-af5f-cd16fb20bbde"). InnerVolumeSpecName "kube-api-access-hlqsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:08.860997 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.860958    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:29:08.860997 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.860993    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:29:08.860997 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.861003    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlqsk\" (UniqueName: \"kubernetes.io/projected/8274001c-2f72-4782-af5f-cd16fb20bbde-kube-api-access-hlqsk\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:29:08.861238 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.861012    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:29:08.861238 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.861020    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8274001c-2f72-4782-af5f-cd16fb20bbde-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:29:08.861238 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:08.861030    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8274001c-2f72-4782-af5f-cd16fb20bbde-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:29:09.299115 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.299079    4334 generic.go:356] "Generic (PLEG): container finished" podID="8274001c-2f72-4782-af5f-cd16fb20bbde" containerID="148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3" exitCode=0
Oct 30 03:29:09.299287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.299127    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.299287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.299163    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8274001c-2f72-4782-af5f-cd16fb20bbde","Type":"ContainerDied","Data":"148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3"}
Oct 30 03:29:09.299287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.299222    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8274001c-2f72-4782-af5f-cd16fb20bbde","Type":"ContainerDied","Data":"b652276ab63b6aa6208e105d1320734152c379926f6060803ba330cfe449f471"}
Oct 30 03:29:09.299287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.299246    4334 scope.go:122] "RemoveContainer" containerID="148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3"
Oct 30 03:29:09.308826 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.308802    4334 scope.go:122] "RemoveContainer" containerID="148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3"
Oct 30 03:29:09.309282 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:29:09.309253    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3\": container with ID starting with 148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3 not found: ID does not exist" containerID="148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3"
Oct 30 03:29:09.309376 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.309290    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3"} err="failed to get container status \"148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3\": rpc error: code = NotFound desc = could not find container \"148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3\": container with ID starting with 148db7976fd503c0519dcbe365a0d8bf4347f9ac1050ee93ae892cebe85d12d3 not found: ID does not exist"
Oct 30 03:29:09.347250 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.347223    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:09.359335 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.359301    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:09.393277 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.393248    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:09.393645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.393629    4334 cpu_manager.go:414] "containerMap: removing container" podUID="8274001c-2f72-4782-af5f-cd16fb20bbde" containerName="otc-container"
Oct 30 03:29:09.393645 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.393644    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8274001c-2f72-4782-af5f-cd16fb20bbde" containerName="otc-container"
Oct 30 03:29:09.393780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.393665    4334 cpu_manager.go:414] "containerMap: removing container" podUID="60e290ab-2016-4748-811e-7cc28c0e28af" containerName="tenant-metrics-exporter"
Oct 30 03:29:09.393780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.393673    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e290ab-2016-4748-811e-7cc28c0e28af" containerName="tenant-metrics-exporter"
Oct 30 03:29:09.393843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.393781    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="8274001c-2f72-4782-af5f-cd16fb20bbde" containerName="otc-container"
Oct 30 03:29:09.393843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.393794    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e290ab-2016-4748-811e-7cc28c0e28af" containerName="tenant-metrics-exporter"
Oct 30 03:29:09.396725 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.396704    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.417930 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.417871    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:29:09.418051 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.417948    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:29:09.418051 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.417995    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:29:09.419107 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.419083    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:09.427980 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.427960    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:29:09.427980 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.427954    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-db539ca7\""
Oct 30 03:29:09.428134 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.428041    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:29:09.567081 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.567048    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.567081 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.567082    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.567299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.567234    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sld2z\" (UniqueName: \"kubernetes.io/projected/705033d6-25c2-48d7-86f7-0e7f599fe0a5-kube-api-access-sld2z\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.567299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.567286    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.567377 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.567328    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.567412 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.567391    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.668813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.668718    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.668813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.668758    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.668813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.668800    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.669433 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.668819    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.669433 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.668859    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sld2z\" (UniqueName: \"kubernetes.io/projected/705033d6-25c2-48d7-86f7-0e7f599fe0a5-kube-api-access-sld2z\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.669433 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.668875    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.669663 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.669638    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.671414 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.671388    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.671529 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.671484    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.671529 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.671486    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.671602 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.671531    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.688234 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.688210    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sld2z\" (UniqueName: \"kubernetes.io/projected/705033d6-25c2-48d7-86f7-0e7f599fe0a5-kube-api-access-sld2z\") pod \"data-science-collector-collector-1\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:09.708108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:09.708088    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:10.008041 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:10.007964    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:10.012686 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:29:10.012653    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705033d6_25c2_48d7_86f7_0e7f599fe0a5.slice/crio-15d8fe289b573d76120d30d728a750eae94e32d136fd24258099866014b99d74 WatchSource:0}: Error finding container 15d8fe289b573d76120d30d728a750eae94e32d136fd24258099866014b99d74: Status 404 returned error can't find the container with id 15d8fe289b573d76120d30d728a750eae94e32d136fd24258099866014b99d74
Oct 30 03:29:10.014838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:10.014802    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:29:10.014924 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:10.014864    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:29:10.307370 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:10.307290    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"705033d6-25c2-48d7-86f7-0e7f599fe0a5","Type":"ContainerStarted","Data":"3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324"}
Oct 30 03:29:10.307370 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:10.307337    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"705033d6-25c2-48d7-86f7-0e7f599fe0a5","Type":"ContainerStarted","Data":"15d8fe289b573d76120d30d728a750eae94e32d136fd24258099866014b99d74"}
Oct 30 03:29:10.355842 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:10.355772    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.355749294 podStartE2EDuration="1.355749294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:10.343638191 +0000 UTC m=+2829.350228445" watchObservedRunningTime="2026-10-30 03:29:10.355749294 +0000 UTC m=+2829.362339543"
Oct 30 03:29:11.158632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:29:11.158598    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8274001c-2f72-4782-af5f-cd16fb20bbde" path="/var/lib/kubelet/pods/8274001c-2f72-4782-af5f-cd16fb20bbde/volumes"
Oct 30 03:29:20.978366 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:29:20.978291    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:29:50.977362 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:29:50.977319    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:30:20.977576 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:30:20.977542    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:30:50.977774 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:30:50.977686    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:31:20.977571 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:31:20.977536    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:31:28.303166 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:28.303128    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:28.303836 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:28.303359    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="705033d6-25c2-48d7-86f7-0e7f599fe0a5" containerName="otc-container" containerID="cri-o://3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324" gracePeriod=30
Oct 30 03:31:30.656186 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.656162    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:30.707697 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.707635    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otc-internal\") pod \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") "
Oct 30 03:31:30.707843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.707711    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otlp-tls-certs\") pod \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") "
Oct 30 03:31:30.707843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.707742    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-data-science-collector-ta-client-cert\") pod \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") "
Oct 30 03:31:30.707843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.707784    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705033d6-25c2-48d7-86f7-0e7f599fe0a5-kube-api-access-sld2z\" (UniqueName: \"kubernetes.io/projected/705033d6-25c2-48d7-86f7-0e7f599fe0a5-kube-api-access-sld2z\") pod \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") "
Oct 30 03:31:30.707843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.707808    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-tls-certs\") pod \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") "
Oct 30 03:31:30.708078 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.707892    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-monitoring-tls-certs\") pod \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\" (UID: \"705033d6-25c2-48d7-86f7-0e7f599fe0a5\") "
Oct 30 03:31:30.708378 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.708089    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otc-internal" pod "705033d6-25c2-48d7-86f7-0e7f599fe0a5" (UID: "705033d6-25c2-48d7-86f7-0e7f599fe0a5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:30.708378 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.708180    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.710200 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.710176    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-monitoring-tls-certs" pod "705033d6-25c2-48d7-86f7-0e7f599fe0a5" (UID: "705033d6-25c2-48d7-86f7-0e7f599fe0a5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:30.710559 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.710522    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otlp-tls-certs" pod "705033d6-25c2-48d7-86f7-0e7f599fe0a5" (UID: "705033d6-25c2-48d7-86f7-0e7f599fe0a5"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:30.710767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.710741    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-tls-certs" pod "705033d6-25c2-48d7-86f7-0e7f599fe0a5" (UID: "705033d6-25c2-48d7-86f7-0e7f599fe0a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:30.711571 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.711545    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705033d6-25c2-48d7-86f7-0e7f599fe0a5-kube-api-access-sld2z" pod "705033d6-25c2-48d7-86f7-0e7f599fe0a5" (UID: "705033d6-25c2-48d7-86f7-0e7f599fe0a5"). InnerVolumeSpecName "kube-api-access-sld2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:30.711783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.711756    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-data-science-collector-ta-client-cert" pod "705033d6-25c2-48d7-86f7-0e7f599fe0a5" (UID: "705033d6-25c2-48d7-86f7-0e7f599fe0a5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:30.809355 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.809331    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.809355 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.809352    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.809544 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.809363    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sld2z\" (UniqueName: \"kubernetes.io/projected/705033d6-25c2-48d7-86f7-0e7f599fe0a5-kube-api-access-sld2z\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.809544 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.809373    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.809544 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:30.809383    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/705033d6-25c2-48d7-86f7-0e7f599fe0a5-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:31.001500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.001404    4334 generic.go:356] "Generic (PLEG): container finished" podID="705033d6-25c2-48d7-86f7-0e7f599fe0a5" containerID="3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324" exitCode=0
Oct 30 03:31:31.001500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.001462    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.001500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.001464    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"705033d6-25c2-48d7-86f7-0e7f599fe0a5","Type":"ContainerDied","Data":"3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324"}
Oct 30 03:31:31.001720 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.001510    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"705033d6-25c2-48d7-86f7-0e7f599fe0a5","Type":"ContainerDied","Data":"15d8fe289b573d76120d30d728a750eae94e32d136fd24258099866014b99d74"}
Oct 30 03:31:31.001720 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.001532    4334 scope.go:122] "RemoveContainer" containerID="3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324"
Oct 30 03:31:31.012816 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.012796    4334 scope.go:122] "RemoveContainer" containerID="3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324"
Oct 30 03:31:31.013110 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:31:31.013093    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324\": container with ID starting with 3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324 not found: ID does not exist" containerID="3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324"
Oct 30 03:31:31.013184 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.013117    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324"} err="failed to get container status \"3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324\": rpc error: code = NotFound desc = could not find container \"3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324\": container with ID starting with 3927ab9b75c8d6453c14fcea2a6c167dd0586ade66e471d23395b7682d779324 not found: ID does not exist"
Oct 30 03:31:31.049765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.049742    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:31.062981 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.061906    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:31.092523 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.092497    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:31.092895 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.092878    4334 cpu_manager.go:414] "containerMap: removing container" podUID="705033d6-25c2-48d7-86f7-0e7f599fe0a5" containerName="otc-container"
Oct 30 03:31:31.092945 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.092897    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705033d6-25c2-48d7-86f7-0e7f599fe0a5" containerName="otc-container"
Oct 30 03:31:31.093000 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.092987    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="705033d6-25c2-48d7-86f7-0e7f599fe0a5" containerName="otc-container"
Oct 30 03:31:31.097366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.097343    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.112019 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.111996    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:31:31.112252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.112227    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:31:31.112364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.112326    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:31:31.112364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.112344    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:31:31.112500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.112474    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:31:31.112600 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.112576    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:31:31.114776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.114757    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:31.158591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.158561    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705033d6-25c2-48d7-86f7-0e7f599fe0a5" path="/var/lib/kubelet/pods/705033d6-25c2-48d7-86f7-0e7f599fe0a5/volumes"
Oct 30 03:31:31.212204 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.212180    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.212327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.212212    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.212327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.212236    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.212417 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.212371    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.212417 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.212407    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6eba85f-d0d7-4f44-a228-8753b9550764-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.212507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.212424    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4ps\" (UniqueName: \"kubernetes.io/projected/c6eba85f-d0d7-4f44-a228-8753b9550764-kube-api-access-4g4ps\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.313924 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.313844    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.313924 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.313895    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6eba85f-d0d7-4f44-a228-8753b9550764-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.314137 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.314005    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4ps\" (UniqueName: \"kubernetes.io/projected/c6eba85f-d0d7-4f44-a228-8753b9550764-kube-api-access-4g4ps\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.314137 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.314059    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.314236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.314180    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.314288 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.314235    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.314701 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.314673    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6eba85f-d0d7-4f44-a228-8753b9550764-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.316386 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.316362    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.316559 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.316539    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.316768 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.316751    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.316813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.316796    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.331458 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.331426    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4ps\" (UniqueName: \"kubernetes.io/projected/c6eba85f-d0d7-4f44-a228-8753b9550764-kube-api-access-4g4ps\") pod \"data-science-collector-collector-1\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.409847 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.409824    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:31.704518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.704470    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:31.708568 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:31:31.708535    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eba85f_d0d7_4f44_a228_8753b9550764.slice/crio-8c61aebca57eda486c56298c59245806bb5c4f25cf644fd9054a52705f4a1f67 WatchSource:0}: Error finding container 8c61aebca57eda486c56298c59245806bb5c4f25cf644fd9054a52705f4a1f67: Status 404 returned error can't find the container with id 8c61aebca57eda486c56298c59245806bb5c4f25cf644fd9054a52705f4a1f67
Oct 30 03:31:31.710883 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.710842    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:31:31.710977 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:31.710921    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:31:32.008983 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:32.008902    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c6eba85f-d0d7-4f44-a228-8753b9550764","Type":"ContainerStarted","Data":"94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f"}
Oct 30 03:31:32.008983 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:32.008956    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c6eba85f-d0d7-4f44-a228-8753b9550764","Type":"ContainerStarted","Data":"8c61aebca57eda486c56298c59245806bb5c4f25cf644fd9054a52705f4a1f67"}
Oct 30 03:31:32.050088 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:32.050033    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.050018123 podStartE2EDuration="1.050018123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:32.048142691 +0000 UTC m=+2971.054732946" watchObservedRunningTime="2026-10-30 03:31:32.050018123 +0000 UTC m=+2971.056608376"
Oct 30 03:31:47.469430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.469394    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:47.469943 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.469622    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c6eba85f-d0d7-4f44-a228-8753b9550764" containerName="otc-container" containerID="cri-o://94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f" gracePeriod=30
Oct 30 03:31:47.714774 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.714748    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:47.846779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.846692    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-tls-certs\") pod \"c6eba85f-d0d7-4f44-a228-8753b9550764\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") "
Oct 30 03:31:47.846955 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.846822    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-data-science-collector-ta-client-cert\") pod \"c6eba85f-d0d7-4f44-a228-8753b9550764\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") "
Oct 30 03:31:47.846955 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.846908    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6eba85f-d0d7-4f44-a228-8753b9550764-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6eba85f-d0d7-4f44-a228-8753b9550764-otc-internal\") pod \"c6eba85f-d0d7-4f44-a228-8753b9550764\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") "
Oct 30 03:31:47.847075 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.846969    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6eba85f-d0d7-4f44-a228-8753b9550764-kube-api-access-4g4ps\" (UniqueName: \"kubernetes.io/projected/c6eba85f-d0d7-4f44-a228-8753b9550764-kube-api-access-4g4ps\") pod \"c6eba85f-d0d7-4f44-a228-8753b9550764\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") "
Oct 30 03:31:47.847132 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.847080    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-otlp-tls-certs\") pod \"c6eba85f-d0d7-4f44-a228-8753b9550764\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") "
Oct 30 03:31:47.847132 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.847112    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-monitoring-tls-certs\") pod \"c6eba85f-d0d7-4f44-a228-8753b9550764\" (UID: \"c6eba85f-d0d7-4f44-a228-8753b9550764\") "
Oct 30 03:31:47.847342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.847308    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eba85f-d0d7-4f44-a228-8753b9550764-otc-internal" pod "c6eba85f-d0d7-4f44-a228-8753b9550764" (UID: "c6eba85f-d0d7-4f44-a228-8753b9550764"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:47.847432 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.847394    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6eba85f-d0d7-4f44-a228-8753b9550764-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:47.849085 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.849054    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-data-science-collector-ta-client-cert" pod "c6eba85f-d0d7-4f44-a228-8753b9550764" (UID: "c6eba85f-d0d7-4f44-a228-8753b9550764"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:47.849207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.849130    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-otlp-tls-certs" pod "c6eba85f-d0d7-4f44-a228-8753b9550764" (UID: "c6eba85f-d0d7-4f44-a228-8753b9550764"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:47.849372 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.849352    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6eba85f-d0d7-4f44-a228-8753b9550764-kube-api-access-4g4ps" pod "c6eba85f-d0d7-4f44-a228-8753b9550764" (UID: "c6eba85f-d0d7-4f44-a228-8753b9550764"). InnerVolumeSpecName "kube-api-access-4g4ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:47.849483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.849380    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-tls-certs" pod "c6eba85f-d0d7-4f44-a228-8753b9550764" (UID: "c6eba85f-d0d7-4f44-a228-8753b9550764"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:47.849483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.849399    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-monitoring-tls-certs" pod "c6eba85f-d0d7-4f44-a228-8753b9550764" (UID: "c6eba85f-d0d7-4f44-a228-8753b9550764"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:47.948508 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.948481    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4ps\" (UniqueName: \"kubernetes.io/projected/c6eba85f-d0d7-4f44-a228-8753b9550764-kube-api-access-4g4ps\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:47.948508 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.948502    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:47.948508 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.948512    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:47.948722 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.948522    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:47.948722 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:47.948531    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6eba85f-d0d7-4f44-a228-8753b9550764-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:31:48.087503 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.087442    4334 generic.go:356] "Generic (PLEG): container finished" podID="c6eba85f-d0d7-4f44-a228-8753b9550764" containerID="94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f" exitCode=0
Oct 30 03:31:48.087673 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.087511    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.087673 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.087544    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c6eba85f-d0d7-4f44-a228-8753b9550764","Type":"ContainerDied","Data":"94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f"}
Oct 30 03:31:48.087673 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.087601    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c6eba85f-d0d7-4f44-a228-8753b9550764","Type":"ContainerDied","Data":"8c61aebca57eda486c56298c59245806bb5c4f25cf644fd9054a52705f4a1f67"}
Oct 30 03:31:48.087673 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.087623    4334 scope.go:122] "RemoveContainer" containerID="94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f"
Oct 30 03:31:48.097443 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.097427    4334 scope.go:122] "RemoveContainer" containerID="94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f"
Oct 30 03:31:48.097719 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:31:48.097701    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f\": container with ID starting with 94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f not found: ID does not exist" containerID="94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f"
Oct 30 03:31:48.097771 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.097726    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f"} err="failed to get container status \"94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f\": rpc error: code = NotFound desc = could not find container \"94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f\": container with ID starting with 94ecbab1cfaa78263560d00f8bde58ea2b478e47c164b075af3fbd8f5dcdab1f not found: ID does not exist"
Oct 30 03:31:48.136633 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.136602    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:48.147881 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.147858    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:48.173746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.173721    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:48.174126 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.174111    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c6eba85f-d0d7-4f44-a228-8753b9550764" containerName="otc-container"
Oct 30 03:31:48.174181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.174128    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6eba85f-d0d7-4f44-a228-8753b9550764" containerName="otc-container"
Oct 30 03:31:48.174219 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.174202    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6eba85f-d0d7-4f44-a228-8753b9550764" containerName="otc-container"
Oct 30 03:31:48.178764 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.178749    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.195019 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.194997    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:48.196482 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.196415    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:31:48.196624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.196520    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:31:48.196624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.196567    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:31:48.196624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.196611    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:31:48.196770 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.196653    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:31:48.277705 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.277683    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:31:48.351810 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.351729    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.351810 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.351768    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.351810 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.351805    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.352047 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.351843    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lbhf\" (UniqueName: \"kubernetes.io/projected/a39ec965-06a3-4184-a58c-2d33e228d614-kube-api-access-6lbhf\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.352047 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.351984    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a39ec965-06a3-4184-a58c-2d33e228d614-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.352047 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.352001    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.453114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.453086    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a39ec965-06a3-4184-a58c-2d33e228d614-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.453114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.453113    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.453330 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.453145    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.453330 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.453171    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.453330 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.453212    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.453330 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.453229    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lbhf\" (UniqueName: \"kubernetes.io/projected/a39ec965-06a3-4184-a58c-2d33e228d614-kube-api-access-6lbhf\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.453877 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.453847    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a39ec965-06a3-4184-a58c-2d33e228d614-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.455821 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.455792    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.455952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.455828    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.455952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.455908    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.456064 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.455984    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.471088 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.471068    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lbhf\" (UniqueName: \"kubernetes.io/projected/a39ec965-06a3-4184-a58c-2d33e228d614-kube-api-access-6lbhf\") pod \"data-science-collector-collector-1\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.490995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.490976    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:31:48.791502 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.791467    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:31:48.795737 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:31:48.795709    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39ec965_06a3_4184_a58c_2d33e228d614.slice/crio-89e93805a82d13107a5dadf25ffb78d7b093a9078b266cbf3cadd3314ce2fe73 WatchSource:0}: Error finding container 89e93805a82d13107a5dadf25ffb78d7b093a9078b266cbf3cadd3314ce2fe73: Status 404 returned error can't find the container with id 89e93805a82d13107a5dadf25ffb78d7b093a9078b266cbf3cadd3314ce2fe73
Oct 30 03:31:48.798053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.798022    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:31:48.798132 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:48.798080    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:31:49.095624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:49.095546    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a39ec965-06a3-4184-a58c-2d33e228d614","Type":"ContainerStarted","Data":"6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c"}
Oct 30 03:31:49.095624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:49.095595    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a39ec965-06a3-4184-a58c-2d33e228d614","Type":"ContainerStarted","Data":"89e93805a82d13107a5dadf25ffb78d7b093a9078b266cbf3cadd3314ce2fe73"}
Oct 30 03:31:49.137895 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:49.137804    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.137789529 podStartE2EDuration="1.137789529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:49.136168187 +0000 UTC m=+2988.142758441" watchObservedRunningTime="2026-10-30 03:31:49.137789529 +0000 UTC m=+2988.144379782"
Oct 30 03:31:49.173037 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:31:49.171637    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6eba85f-d0d7-4f44-a228-8753b9550764" path="/var/lib/kubelet/pods/c6eba85f-d0d7-4f44-a228-8753b9550764/volumes"
Oct 30 03:31:50.979337 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:31:50.979303    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:32:20.978726 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:32:20.978645    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:32:50.978002 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:32:50.977965    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:33:20.977526 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:33:20.977485    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:33:50.977936 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:33:50.977853    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:34:20.977883 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:34:20.977842    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:34:50.977735 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:34:50.977696    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:35:20.977848 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:35:20.977759    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:35:50.978221 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:35:50.978180    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:36:18.596691 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:18.596660    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:36:18.597079 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:18.596867    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a39ec965-06a3-4184-a58c-2d33e228d614" containerName="otc-container" containerID="cri-o://6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c" gracePeriod=30
Oct 30 03:36:20.834054 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.834031    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:20.904843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.904813    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39ec965-06a3-4184-a58c-2d33e228d614-kube-api-access-6lbhf\" (UniqueName: \"kubernetes.io/projected/a39ec965-06a3-4184-a58c-2d33e228d614-kube-api-access-6lbhf\") pod \"a39ec965-06a3-4184-a58c-2d33e228d614\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") "
Oct 30 03:36:20.905003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.904861    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39ec965-06a3-4184-a58c-2d33e228d614-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a39ec965-06a3-4184-a58c-2d33e228d614-otc-internal\") pod \"a39ec965-06a3-4184-a58c-2d33e228d614\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") "
Oct 30 03:36:20.905003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.904889    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-otlp-tls-certs\") pod \"a39ec965-06a3-4184-a58c-2d33e228d614\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") "
Oct 30 03:36:20.905003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.904975    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-monitoring-tls-certs\") pod \"a39ec965-06a3-4184-a58c-2d33e228d614\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") "
Oct 30 03:36:20.905176 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.905004    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-tls-certs\") pod \"a39ec965-06a3-4184-a58c-2d33e228d614\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") "
Oct 30 03:36:20.905176 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.905056    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-data-science-collector-ta-client-cert\") pod \"a39ec965-06a3-4184-a58c-2d33e228d614\" (UID: \"a39ec965-06a3-4184-a58c-2d33e228d614\") "
Oct 30 03:36:20.905855 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.905804    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39ec965-06a3-4184-a58c-2d33e228d614-otc-internal" pod "a39ec965-06a3-4184-a58c-2d33e228d614" (UID: "a39ec965-06a3-4184-a58c-2d33e228d614"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:36:20.907867 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.907817    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-monitoring-tls-certs" pod "a39ec965-06a3-4184-a58c-2d33e228d614" (UID: "a39ec965-06a3-4184-a58c-2d33e228d614"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:20.908007 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.907965    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39ec965-06a3-4184-a58c-2d33e228d614-kube-api-access-6lbhf" pod "a39ec965-06a3-4184-a58c-2d33e228d614" (UID: "a39ec965-06a3-4184-a58c-2d33e228d614"). InnerVolumeSpecName "kube-api-access-6lbhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:36:20.908259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.908225    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-tls-certs" pod "a39ec965-06a3-4184-a58c-2d33e228d614" (UID: "a39ec965-06a3-4184-a58c-2d33e228d614"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:20.914116 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.914083    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-otlp-tls-certs" pod "a39ec965-06a3-4184-a58c-2d33e228d614" (UID: "a39ec965-06a3-4184-a58c-2d33e228d614"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:20.914217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:20.914101    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-data-science-collector-ta-client-cert" pod "a39ec965-06a3-4184-a58c-2d33e228d614" (UID: "a39ec965-06a3-4184-a58c-2d33e228d614"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:36:20.977847 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:36:20.977777    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:36:21.009384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.009357    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:36:21.009384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.009379    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:36:21.009585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.009392    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:36:21.009585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.009401    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lbhf\" (UniqueName: \"kubernetes.io/projected/a39ec965-06a3-4184-a58c-2d33e228d614-kube-api-access-6lbhf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:36:21.009585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.009411    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a39ec965-06a3-4184-a58c-2d33e228d614-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:36:21.009585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.009419    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a39ec965-06a3-4184-a58c-2d33e228d614-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:36:21.362267 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.362188    4334 generic.go:356] "Generic (PLEG): container finished" podID="a39ec965-06a3-4184-a58c-2d33e228d614" containerID="6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c" exitCode=0
Oct 30 03:36:21.362267 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.362258    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.362457 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.362270    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a39ec965-06a3-4184-a58c-2d33e228d614","Type":"ContainerDied","Data":"6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c"}
Oct 30 03:36:21.362457 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.362332    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a39ec965-06a3-4184-a58c-2d33e228d614","Type":"ContainerDied","Data":"89e93805a82d13107a5dadf25ffb78d7b093a9078b266cbf3cadd3314ce2fe73"}
Oct 30 03:36:21.362457 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.362355    4334 scope.go:122] "RemoveContainer" containerID="6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c"
Oct 30 03:36:21.370594 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.370573    4334 scope.go:122] "RemoveContainer" containerID="6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c"
Oct 30 03:36:21.370856 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:36:21.370838    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c\": container with ID starting with 6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c not found: ID does not exist" containerID="6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c"
Oct 30 03:36:21.370905 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.370865    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c"} err="failed to get container status \"6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c\": rpc error: code = NotFound desc = could not find container \"6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c\": container with ID starting with 6e82259ba2435b3e39d3eb6fa22e4cf41fa0592549201989a814044678fe977c not found: ID does not exist"
Oct 30 03:36:21.416106 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.416077    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:36:21.429248 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.429215    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:36:21.458689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.458667    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:36:21.459165 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.459149    4334 cpu_manager.go:414] "containerMap: removing container" podUID="a39ec965-06a3-4184-a58c-2d33e228d614" containerName="otc-container"
Oct 30 03:36:21.459210 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.459170    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39ec965-06a3-4184-a58c-2d33e228d614" containerName="otc-container"
Oct 30 03:36:21.459303 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.459291    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39ec965-06a3-4184-a58c-2d33e228d614" containerName="otc-container"
Oct 30 03:36:21.464108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.464092    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.478588 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.478567    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:36:21.479970 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.479936    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:36:21.480072 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.479988    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rzsdv\""
Oct 30 03:36:21.480072 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.480054    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:36:21.490042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.490018    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:36:21.490128 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.490059    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:36:21.490128 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.490081    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:36:21.613765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.613692    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.613765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.613729    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5m2p\" (UniqueName: \"kubernetes.io/projected/ec082f00-8899-45c3-9b20-0e3868741fb6-kube-api-access-q5m2p\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.613765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.613749    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.613953 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.613807    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.613953 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.613867    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.613953 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.613897    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec082f00-8899-45c3-9b20-0e3868741fb6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.714615 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.714583    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.714615 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.714618    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5m2p\" (UniqueName: \"kubernetes.io/projected/ec082f00-8899-45c3-9b20-0e3868741fb6-kube-api-access-q5m2p\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.714808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.714635    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.714808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.714653    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.714808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.714679    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.714956 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.714813    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec082f00-8899-45c3-9b20-0e3868741fb6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.715547 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.715520    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec082f00-8899-45c3-9b20-0e3868741fb6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.717366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.717334    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.717493 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.717375    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.717493 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.717391    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.717493 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.717377    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.734392 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.734370    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5m2p\" (UniqueName: \"kubernetes.io/projected/ec082f00-8899-45c3-9b20-0e3868741fb6-kube-api-access-q5m2p\") pod \"data-science-collector-collector-1\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:21.776334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:21.776306    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:36:22.079226 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:22.079192    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:36:22.081849 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:36:22.081823    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec082f00_8899_45c3_9b20_0e3868741fb6.slice/crio-13805ee7570079dfc8cbf88e0aaa16f300b317134bf09d7def34ccfcff35d503 WatchSource:0}: Error finding container 13805ee7570079dfc8cbf88e0aaa16f300b317134bf09d7def34ccfcff35d503: Status 404 returned error can't find the container with id 13805ee7570079dfc8cbf88e0aaa16f300b317134bf09d7def34ccfcff35d503
Oct 30 03:36:22.083896 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:22.083867    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:36:22.083959 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:22.083926    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:36:22.370043 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:22.369969    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec082f00-8899-45c3-9b20-0e3868741fb6","Type":"ContainerStarted","Data":"e49fbfc3d4dc36e69a780d42e61a061c75433462143f56ec24e94a75db99b3da"}
Oct 30 03:36:22.370043 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:22.370014    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec082f00-8899-45c3-9b20-0e3868741fb6","Type":"ContainerStarted","Data":"13805ee7570079dfc8cbf88e0aaa16f300b317134bf09d7def34ccfcff35d503"}
Oct 30 03:36:23.159271 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:23.159239    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39ec965-06a3-4184-a58c-2d33e228d614" path="/var/lib/kubelet/pods/a39ec965-06a3-4184-a58c-2d33e228d614/volumes"
Oct 30 03:36:42.616897 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:42.616842    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=21.616827262 podStartE2EDuration="21.616827262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:22.412031117 +0000 UTC m=+3261.418621371" watchObservedRunningTime="2026-10-30 03:36:42.616827262 +0000 UTC m=+3281.623417515"
Oct 30 03:36:42.618299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:42.618273    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"]
Oct 30 03:36:42.621784 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:42.621764    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"
Oct 30 03:36:42.636555 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:42.636515    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"]
Oct 30 03:36:42.648697 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:42.648671    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-fhc4j\""
Oct 30 03:36:42.652891 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:42.652871    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"
Oct 30 03:36:42.951624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:42.951600    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"]
Oct 30 03:36:42.954079 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:36:42.954043    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff1316ee_a718_4fbe_8842_c1916cdbb639.slice/crio-50ef775558f942e3ba1d12085379cc1cb6b6db3bcd0477c8a0c30f17f277c114 WatchSource:0}: Error finding container 50ef775558f942e3ba1d12085379cc1cb6b6db3bcd0477c8a0c30f17f277c114: Status 404 returned error can't find the container with id 50ef775558f942e3ba1d12085379cc1cb6b6db3bcd0477c8a0c30f17f277c114
Oct 30 03:36:43.469290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:43.469253    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e" event={"ID":"ff1316ee-a718-4fbe-8842-c1916cdbb639","Type":"ContainerStarted","Data":"2639a1f9ef4f9ff3421bb2e5353d0155afe9d5b1bf2708259a42a7af078ee086"}
Oct 30 03:36:43.469473 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:43.469303    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e" event={"ID":"ff1316ee-a718-4fbe-8842-c1916cdbb639","Type":"ContainerStarted","Data":"50ef775558f942e3ba1d12085379cc1cb6b6db3bcd0477c8a0c30f17f277c114"}
Oct 30 03:36:43.515673 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:43.515627    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e" podStartSLOduration=1.515613761 podStartE2EDuration="1.515613761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:43.514307664 +0000 UTC m=+3282.520897915" watchObservedRunningTime="2026-10-30 03:36:43.515613761 +0000 UTC m=+3282.522204016"
Oct 30 03:36:50.978334 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:36:50.978297    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:36:53.517106 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:53.517073    4334 generic.go:356] "Generic (PLEG): container finished" podID="ff1316ee-a718-4fbe-8842-c1916cdbb639" containerID="2639a1f9ef4f9ff3421bb2e5353d0155afe9d5b1bf2708259a42a7af078ee086" exitCode=0
Oct 30 03:36:53.517542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:53.517144    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e" event={"ID":"ff1316ee-a718-4fbe-8842-c1916cdbb639","Type":"ContainerDied","Data":"2639a1f9ef4f9ff3421bb2e5353d0155afe9d5b1bf2708259a42a7af078ee086"}
Oct 30 03:36:54.653646 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:54.653623    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"
Oct 30 03:36:55.528109 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:55.528038    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"
Oct 30 03:36:55.528109 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:55.528046    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e" event={"ID":"ff1316ee-a718-4fbe-8842-c1916cdbb639","Type":"ContainerDied","Data":"50ef775558f942e3ba1d12085379cc1cb6b6db3bcd0477c8a0c30f17f277c114"}
Oct 30 03:36:55.528109 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:36:55.528078    4334 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50ef775558f942e3ba1d12085379cc1cb6b6db3bcd0477c8a0c30f17f277c114"
Oct 30 03:37:20.978172 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:37:20.978133    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:37:50.979065 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:37:50.979029    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:38:20.978678 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:38:20.978568    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:38:50.978199 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:38:50.978163    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:39:20.977299 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:39:20.977262    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:39:50.978051 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:39:50.977969    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:40:11.657133 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:11.657094    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"]
Oct 30 03:40:11.661531 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:11.661507    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e33168bbbd8d7e"]
Oct 30 03:40:13.159026 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:13.158993    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1316ee-a718-4fbe-8842-c1916cdbb639" path="/var/lib/kubelet/pods/ff1316ee-a718-4fbe-8842-c1916cdbb639/volumes"
Oct 30 03:40:20.977156 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:20.977116    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:40:41.176891 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.176858    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"]
Oct 30 03:40:41.177493 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.177082    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerName="kube-rbac-proxy" containerID="cri-o://4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699" gracePeriod=30
Oct 30 03:40:41.177493 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.177150    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerName="prom-label-proxy" containerID="cri-o://145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045" gracePeriod=30
Oct 30 03:40:41.198098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.198076    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"]
Oct 30 03:40:41.198354 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.198325    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerName="kube-rbac-proxy" containerID="cri-o://042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8" gracePeriod=30
Oct 30 03:40:41.198354 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.198354    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerName="prom-label-proxy" containerID="cri-o://dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d" gracePeriod=30
Oct 30 03:40:41.442294 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.442233    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:40:41.460096 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.460072    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:40:41.551659 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.551625    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-prometheus-tls-ca\") pod \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") "
Oct 30 03:40:41.551841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.551695    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-client-ca\") pod \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") "
Oct 30 03:40:41.551841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.551727    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01a2d62-bcab-408e-acd7-7e875276d0e0-kube-api-access-n4cw7\" (UniqueName: \"kubernetes.io/projected/b01a2d62-bcab-408e-acd7-7e875276d0e0-kube-api-access-n4cw7\") pod \"b01a2d62-bcab-408e-acd7-7e875276d0e0\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") "
Oct 30 03:40:41.551841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.551777    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-config\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-config\") pod \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") "
Oct 30 03:40:41.551841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.551818    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-tls\" (UniqueName: \"kubernetes.io/secret/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-tls\") pod \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") "
Oct 30 03:40:41.552070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.551847    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01a2d62-bcab-408e-acd7-7e875276d0e0-config\" (UniqueName: \"kubernetes.io/configmap/b01a2d62-bcab-408e-acd7-7e875276d0e0-config\") pod \"b01a2d62-bcab-408e-acd7-7e875276d0e0\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") "
Oct 30 03:40:41.552070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.551881    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01a2d62-bcab-408e-acd7-7e875276d0e0-tls\" (UniqueName: \"kubernetes.io/secret/b01a2d62-bcab-408e-acd7-7e875276d0e0-tls\") pod \"b01a2d62-bcab-408e-acd7-7e875276d0e0\" (UID: \"b01a2d62-bcab-408e-acd7-7e875276d0e0\") "
Oct 30 03:40:41.552070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.551914    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-kube-api-access-n2vjx\" (UniqueName: \"kubernetes.io/projected/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-kube-api-access-n2vjx\") pod \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\" (UID: \"6aeb82e9-d212-419e-9cc3-24fcf9c91d33\") "
Oct 30 03:40:41.552070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.552030    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-prometheus-tls-ca" pod "6aeb82e9-d212-419e-9cc3-24fcf9c91d33" (UID: "6aeb82e9-d212-419e-9cc3-24fcf9c91d33"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:41.552284 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.552104    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-client-ca" pod "6aeb82e9-d212-419e-9cc3-24fcf9c91d33" (UID: "6aeb82e9-d212-419e-9cc3-24fcf9c91d33"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:41.552541 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.552508    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01a2d62-bcab-408e-acd7-7e875276d0e0-config" pod "b01a2d62-bcab-408e-acd7-7e875276d0e0" (UID: "b01a2d62-bcab-408e-acd7-7e875276d0e0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:41.552635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.552579    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-config" pod "6aeb82e9-d212-419e-9cc3-24fcf9c91d33" (UID: "6aeb82e9-d212-419e-9cc3-24fcf9c91d33"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:41.552994 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.552747    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-prometheus-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.552994 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.552772    4334 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-client-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.555280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.555257    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-tls" pod "6aeb82e9-d212-419e-9cc3-24fcf9c91d33" (UID: "6aeb82e9-d212-419e-9cc3-24fcf9c91d33"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:41.555280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.555264    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01a2d62-bcab-408e-acd7-7e875276d0e0-tls" pod "b01a2d62-bcab-408e-acd7-7e875276d0e0" (UID: "b01a2d62-bcab-408e-acd7-7e875276d0e0"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:41.555469 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.555353    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-kube-api-access-n2vjx" pod "6aeb82e9-d212-419e-9cc3-24fcf9c91d33" (UID: "6aeb82e9-d212-419e-9cc3-24fcf9c91d33"). InnerVolumeSpecName "kube-api-access-n2vjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:41.556166 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.556137    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01a2d62-bcab-408e-acd7-7e875276d0e0-kube-api-access-n4cw7" pod "b01a2d62-bcab-408e-acd7-7e875276d0e0" (UID: "b01a2d62-bcab-408e-acd7-7e875276d0e0"). InnerVolumeSpecName "kube-api-access-n4cw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:41.610105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.609888    4334 generic.go:356] "Generic (PLEG): container finished" podID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerID="145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045" exitCode=0
Oct 30 03:40:41.610105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.609925    4334 generic.go:356] "Generic (PLEG): container finished" podID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerID="4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699" exitCode=0
Oct 30 03:40:41.610105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.609942    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"
Oct 30 03:40:41.610105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.609934    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" event={"ID":"6aeb82e9-d212-419e-9cc3-24fcf9c91d33","Type":"ContainerDied","Data":"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045"}
Oct 30 03:40:41.610105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.610094    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" event={"ID":"6aeb82e9-d212-419e-9cc3-24fcf9c91d33","Type":"ContainerDied","Data":"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699"}
Oct 30 03:40:41.610536 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.610130    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm" event={"ID":"6aeb82e9-d212-419e-9cc3-24fcf9c91d33","Type":"ContainerDied","Data":"91dd55f9961b2cf0b2a513ec1b3fbeb233b53e30aca0f0491606193dc84f175f"}
Oct 30 03:40:41.610536 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.610162    4334 scope.go:122] "RemoveContainer" containerID="145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045"
Oct 30 03:40:41.612469 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.612423    4334 generic.go:356] "Generic (PLEG): container finished" podID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerID="dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d" exitCode=0
Oct 30 03:40:41.612469 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.612458    4334 generic.go:356] "Generic (PLEG): container finished" podID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerID="042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8" exitCode=0
Oct 30 03:40:41.612668 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.612534    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" event={"ID":"b01a2d62-bcab-408e-acd7-7e875276d0e0","Type":"ContainerDied","Data":"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d"}
Oct 30 03:40:41.612668 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.612585    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" event={"ID":"b01a2d62-bcab-408e-acd7-7e875276d0e0","Type":"ContainerDied","Data":"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8"}
Oct 30 03:40:41.612668 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.612624    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4" event={"ID":"b01a2d62-bcab-408e-acd7-7e875276d0e0","Type":"ContainerDied","Data":"d908fbe91f7b751ff5b5e6ddf119665738144703342443d026be080ff0fd103c"}
Oct 30 03:40:41.612836 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.612705    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"
Oct 30 03:40:41.620579 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.620559    4334 scope.go:122] "RemoveContainer" containerID="4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699"
Oct 30 03:40:41.630853 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.630822    4334 scope.go:122] "RemoveContainer" containerID="145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045"
Oct 30 03:40:41.631141 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:41.631112    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045\": container with ID starting with 145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045 not found: ID does not exist" containerID="145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045"
Oct 30 03:40:41.631231 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.631151    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045"} err="failed to get container status \"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045\": rpc error: code = NotFound desc = could not find container \"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045\": container with ID starting with 145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045 not found: ID does not exist"
Oct 30 03:40:41.631231 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.631175    4334 scope.go:122] "RemoveContainer" containerID="4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699"
Oct 30 03:40:41.631920 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:41.631479    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699\": container with ID starting with 4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699 not found: ID does not exist" containerID="4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699"
Oct 30 03:40:41.631920 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.631508    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699"} err="failed to get container status \"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699\": rpc error: code = NotFound desc = could not find container \"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699\": container with ID starting with 4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699 not found: ID does not exist"
Oct 30 03:40:41.631920 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.631525    4334 scope.go:122] "RemoveContainer" containerID="145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045"
Oct 30 03:40:41.632152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.632123    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045"} err="failed to get container status \"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045\": rpc error: code = NotFound desc = could not find container \"145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045\": container with ID starting with 145dbb54aee83a73250145b40d377a7ca4e10b18a88df4614b2425ab85bf9045 not found: ID does not exist"
Oct 30 03:40:41.632152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.632141    4334 scope.go:122] "RemoveContainer" containerID="4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699"
Oct 30 03:40:41.632460 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.632422    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699"} err="failed to get container status \"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699\": rpc error: code = NotFound desc = could not find container \"4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699\": container with ID starting with 4f9dcefde587a8e31ae727b648a14e59e85b14698469d25008fbca7ce0278699 not found: ID does not exist"
Oct 30 03:40:41.632543 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.632474    4334 scope.go:122] "RemoveContainer" containerID="dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d"
Oct 30 03:40:41.642324 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.642302    4334 scope.go:122] "RemoveContainer" containerID="042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8"
Oct 30 03:40:41.650914 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.650880    4334 scope.go:122] "RemoveContainer" containerID="dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d"
Oct 30 03:40:41.651248 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:41.651167    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d\": container with ID starting with dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d not found: ID does not exist" containerID="dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d"
Oct 30 03:40:41.651248 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.651199    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d"} err="failed to get container status \"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d\": rpc error: code = NotFound desc = could not find container \"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d\": container with ID starting with dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d not found: ID does not exist"
Oct 30 03:40:41.651248 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.651219    4334 scope.go:122] "RemoveContainer" containerID="042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8"
Oct 30 03:40:41.651569 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:41.651541    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8\": container with ID starting with 042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8 not found: ID does not exist" containerID="042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8"
Oct 30 03:40:41.651676 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.651579    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8"} err="failed to get container status \"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8\": rpc error: code = NotFound desc = could not find container \"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8\": container with ID starting with 042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8 not found: ID does not exist"
Oct 30 03:40:41.651676 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.651603    4334 scope.go:122] "RemoveContainer" containerID="dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d"
Oct 30 03:40:41.651955 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.651916    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d"} err="failed to get container status \"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d\": rpc error: code = NotFound desc = could not find container \"dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d\": container with ID starting with dbc39e42a1bdfd679b7a7fe357ec43a8a0582b9182ec3c3baacbb1ec9021657d not found: ID does not exist"
Oct 30 03:40:41.652064 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.651975    4334 scope.go:122] "RemoveContainer" containerID="042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8"
Oct 30 03:40:41.652305 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.652279    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8"} err="failed to get container status \"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8\": rpc error: code = NotFound desc = could not find container \"042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8\": container with ID starting with 042be5511ca7bf991d962f479302da28eac03fbfc69524b937e3767f9ed4c9a8 not found: ID does not exist"
Oct 30 03:40:41.653760 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.653743    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cw7\" (UniqueName: \"kubernetes.io/projected/b01a2d62-bcab-408e-acd7-7e875276d0e0-kube-api-access-n4cw7\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.653893 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.653762    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.653893 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.653773    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.653893 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.653784    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b01a2d62-bcab-408e-acd7-7e875276d0e0-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.653893 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.653792    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b01a2d62-bcab-408e-acd7-7e875276d0e0-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.653893 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.653801    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vjx\" (UniqueName: \"kubernetes.io/projected/6aeb82e9-d212-419e-9cc3-24fcf9c91d33-kube-api-access-n2vjx\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:41.678869 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.678837    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"]
Oct 30 03:40:41.680655 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.680614    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-tcrpm"]
Oct 30 03:40:41.726338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.726270    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"]
Oct 30 03:40:41.736481 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.736432    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-2lwf4"]
Oct 30 03:40:41.758673 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.758649    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:40:41.758940 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.758893    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="prometheus" containerID="cri-o://dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0" gracePeriod=600
Oct 30 03:40:41.758940 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.758929    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="config-reloader" containerID="cri-o://501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5" gracePeriod=600
Oct 30 03:40:41.759162 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.758937    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="thanos-sidecar" containerID="cri-o://da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55" gracePeriod=600
Oct 30 03:40:41.774311 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.774286    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:40:41.774748 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.774574    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="alertmanager" containerID="cri-o://3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4" gracePeriod=120
Oct 30 03:40:41.774748 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:41.774725    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="config-reloader" containerID="cri-o://33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7" gracePeriod=120
Oct 30 03:40:42.002409 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.002385    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057234    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057301    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-tls-assets\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-tls-assets\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057332    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-kube-api-access-pvfq2\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-kube-api-access-pvfq2\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057357    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-secret-prometheus-web-tls-ca\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057388    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-thanos-prometheus-http-client-file\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057420    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057491    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057602    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057647    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057687    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057727    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057758    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config-out\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.058103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.057793    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config\") pod \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\" (UID: \"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8\") "
Oct 30 03:40:42.060424 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.058623    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-1" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:42.061193 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.061166    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-0" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:42.061598 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.061245    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-2" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:42.062936 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.062896    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-kube-api-access-pvfq2" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "kube-api-access-pvfq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:42.063054 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.062948    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.063682 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.063656    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.063811 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.063748    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config-out" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:42.064147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.064124    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-thanos-prometheus-http-client-file" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.064349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.064326    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.066033 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.065998    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-secret-prometheus-web-tls-ca" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.068094 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.068070    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:40:42.074815 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.074795    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-tls-assets" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:42.077160 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.077141    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config" pod "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" (UID: "c42d6464-fd67-4e6e-a690-7d1c1ea93cc8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.158948 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.158917    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.158948 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.158944    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.158948 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.158955    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvfq2\" (UniqueName: \"kubernetes.io/projected/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-kube-api-access-pvfq2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.158963    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.158973    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.158981    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.158989    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.159018    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.159027    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.159036    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.159045    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.159053    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.159167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.159065    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.170301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.170282    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:40:42.259954 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.259893    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\""
Oct 30 03:40:42.466039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.466007    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:40:42.466265 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.466210    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ec082f00-8899-45c3-9b20-0e3868741fb6" containerName="otc-container" containerID="cri-o://e49fbfc3d4dc36e69a780d42e61a061c75433462143f56ec24e94a75db99b3da" gracePeriod=30
Oct 30 03:40:42.620056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.620022    4334 generic.go:356] "Generic (PLEG): container finished" podID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerID="33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7" exitCode=0
Oct 30 03:40:42.620230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.620096    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"f0b95646-94f7-4f0c-b193-b1a530517b68","Type":"ContainerDied","Data":"33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7"}
Oct 30 03:40:42.622627 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.622558    4334 generic.go:356] "Generic (PLEG): container finished" podID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerID="da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55" exitCode=0
Oct 30 03:40:42.622627 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.622596    4334 generic.go:356] "Generic (PLEG): container finished" podID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerID="501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5" exitCode=0
Oct 30 03:40:42.622627 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.622604    4334 generic.go:356] "Generic (PLEG): container finished" podID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerID="dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0" exitCode=0
Oct 30 03:40:42.622816 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.622669    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:40:42.622991 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.622968    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerDied","Data":"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"}
Oct 30 03:40:42.623108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.623032    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerDied","Data":"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"}
Oct 30 03:40:42.623108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.623058    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerDied","Data":"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"}
Oct 30 03:40:42.623108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.623080    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c42d6464-fd67-4e6e-a690-7d1c1ea93cc8","Type":"ContainerDied","Data":"12ba85d94a5eb7fb0aa4428bf95be148db86b96004634ac4528ef1227ab0cc42"}
Oct 30 03:40:42.623272 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.623118    4334 scope.go:122] "RemoveContainer" containerID="da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"
Oct 30 03:40:42.628388 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.628365    4334 generic.go:356] "Generic (PLEG): container finished" podID="ec082f00-8899-45c3-9b20-0e3868741fb6" containerID="e49fbfc3d4dc36e69a780d42e61a061c75433462143f56ec24e94a75db99b3da" exitCode=0
Oct 30 03:40:42.628646 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.628390    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec082f00-8899-45c3-9b20-0e3868741fb6","Type":"ContainerDied","Data":"e49fbfc3d4dc36e69a780d42e61a061c75433462143f56ec24e94a75db99b3da"}
Oct 30 03:40:42.636236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.636218    4334 scope.go:122] "RemoveContainer" containerID="501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"
Oct 30 03:40:42.643929 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.643906    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.37:9093/-/ready\": dial tcp 10.133.0.37:9093: connect: connection refused"
Oct 30 03:40:42.646186 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.646168    4334 scope.go:122] "RemoveContainer" containerID="dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"
Oct 30 03:40:42.655692 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.655674    4334 scope.go:122] "RemoveContainer" containerID="6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"
Oct 30 03:40:42.665536 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.665511    4334 scope.go:122] "RemoveContainer" containerID="da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"
Oct 30 03:40:42.665827 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:42.665806    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55\": container with ID starting with da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55 not found: ID does not exist" containerID="da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"
Oct 30 03:40:42.665926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.665838    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"} err="failed to get container status \"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55\": rpc error: code = NotFound desc = could not find container \"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55\": container with ID starting with da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55 not found: ID does not exist"
Oct 30 03:40:42.665926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.665859    4334 scope.go:122] "RemoveContainer" containerID="501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"
Oct 30 03:40:42.666140 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:42.666123    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5\": container with ID starting with 501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5 not found: ID does not exist" containerID="501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"
Oct 30 03:40:42.666204 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.666148    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"} err="failed to get container status \"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5\": rpc error: code = NotFound desc = could not find container \"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5\": container with ID starting with 501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5 not found: ID does not exist"
Oct 30 03:40:42.666204 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.666169    4334 scope.go:122] "RemoveContainer" containerID="dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"
Oct 30 03:40:42.666429 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:42.666413    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0\": container with ID starting with dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0 not found: ID does not exist" containerID="dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"
Oct 30 03:40:42.666542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.666435    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"} err="failed to get container status \"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0\": rpc error: code = NotFound desc = could not find container \"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0\": container with ID starting with dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0 not found: ID does not exist"
Oct 30 03:40:42.666588 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.666545    4334 scope.go:122] "RemoveContainer" containerID="6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"
Oct 30 03:40:42.666759 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:42.666742    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7\": container with ID starting with 6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7 not found: ID does not exist" containerID="6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"
Oct 30 03:40:42.666828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.666766    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"} err="failed to get container status \"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7\": rpc error: code = NotFound desc = could not find container \"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7\": container with ID starting with 6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7 not found: ID does not exist"
Oct 30 03:40:42.666828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.666788    4334 scope.go:122] "RemoveContainer" containerID="da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"
Oct 30 03:40:42.667008 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.666990    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"} err="failed to get container status \"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55\": rpc error: code = NotFound desc = could not find container \"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55\": container with ID starting with da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55 not found: ID does not exist"
Oct 30 03:40:42.667049 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667008    4334 scope.go:122] "RemoveContainer" containerID="501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"
Oct 30 03:40:42.667201 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667184    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"} err="failed to get container status \"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5\": rpc error: code = NotFound desc = could not find container \"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5\": container with ID starting with 501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5 not found: ID does not exist"
Oct 30 03:40:42.667245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667201    4334 scope.go:122] "RemoveContainer" containerID="dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"
Oct 30 03:40:42.667374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667357    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"} err="failed to get container status \"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0\": rpc error: code = NotFound desc = could not find container \"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0\": container with ID starting with dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0 not found: ID does not exist"
Oct 30 03:40:42.667412 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667376    4334 scope.go:122] "RemoveContainer" containerID="6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"
Oct 30 03:40:42.667595 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667581    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"} err="failed to get container status \"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7\": rpc error: code = NotFound desc = could not find container \"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7\": container with ID starting with 6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7 not found: ID does not exist"
Oct 30 03:40:42.667643 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667595    4334 scope.go:122] "RemoveContainer" containerID="da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"
Oct 30 03:40:42.667832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667815    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55"} err="failed to get container status \"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55\": rpc error: code = NotFound desc = could not find container \"da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55\": container with ID starting with da7c51b2421361e312b1243295e5bd35e3da40f13b9f2000576f765e48c56f55 not found: ID does not exist"
Oct 30 03:40:42.667832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.667831    4334 scope.go:122] "RemoveContainer" containerID="501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"
Oct 30 03:40:42.668052 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.668031    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5"} err="failed to get container status \"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5\": rpc error: code = NotFound desc = could not find container \"501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5\": container with ID starting with 501930a1d2e13aece94e8e25038a8ee35fbf3acebfc3093b568ee113fcc4d0e5 not found: ID does not exist"
Oct 30 03:40:42.668052 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.668052    4334 scope.go:122] "RemoveContainer" containerID="dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"
Oct 30 03:40:42.668287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.668245    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0"} err="failed to get container status \"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0\": rpc error: code = NotFound desc = could not find container \"dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0\": container with ID starting with dc277545ab21928582348995480fafb3aab6c409ec891394c1e8aea1b38fb8c0 not found: ID does not exist"
Oct 30 03:40:42.668334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.668291    4334 scope.go:122] "RemoveContainer" containerID="6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"
Oct 30 03:40:42.668566 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.668547    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7"} err="failed to get container status \"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7\": rpc error: code = NotFound desc = could not find container \"6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7\": container with ID starting with 6f2b930b1bfc441858810cc26c021b441da9205fd6bf07bd7920b69142a5cfc7 not found: ID does not exist"
Oct 30 03:40:42.680749 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.680726    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:40:42.693997 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.693976    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:40:42.723116 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.723098    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:42.764311 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.764290    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec082f00-8899-45c3-9b20-0e3868741fb6-kube-api-access-q5m2p\" (UniqueName: \"kubernetes.io/projected/ec082f00-8899-45c3-9b20-0e3868741fb6-kube-api-access-q5m2p\") pod \"ec082f00-8899-45c3-9b20-0e3868741fb6\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") "
Oct 30 03:40:42.764430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.764354    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-otlp-tls-certs\") pod \"ec082f00-8899-45c3-9b20-0e3868741fb6\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") "
Oct 30 03:40:42.764430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.764386    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-monitoring-tls-certs\") pod \"ec082f00-8899-45c3-9b20-0e3868741fb6\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") "
Oct 30 03:40:42.764430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.764426    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec082f00-8899-45c3-9b20-0e3868741fb6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec082f00-8899-45c3-9b20-0e3868741fb6-otc-internal\") pod \"ec082f00-8899-45c3-9b20-0e3868741fb6\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") "
Oct 30 03:40:42.764603 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.764482    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-tls-certs\") pod \"ec082f00-8899-45c3-9b20-0e3868741fb6\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") "
Oct 30 03:40:42.764603 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.764514    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-data-science-collector-ta-client-cert\") pod \"ec082f00-8899-45c3-9b20-0e3868741fb6\" (UID: \"ec082f00-8899-45c3-9b20-0e3868741fb6\") "
Oct 30 03:40:42.764933 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.764890    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec082f00-8899-45c3-9b20-0e3868741fb6-otc-internal" pod "ec082f00-8899-45c3-9b20-0e3868741fb6" (UID: "ec082f00-8899-45c3-9b20-0e3868741fb6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:42.767365 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.766889    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec082f00-8899-45c3-9b20-0e3868741fb6-kube-api-access-q5m2p" pod "ec082f00-8899-45c3-9b20-0e3868741fb6" (UID: "ec082f00-8899-45c3-9b20-0e3868741fb6"). InnerVolumeSpecName "kube-api-access-q5m2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:42.767365 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.767012    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-otlp-tls-certs" pod "ec082f00-8899-45c3-9b20-0e3868741fb6" (UID: "ec082f00-8899-45c3-9b20-0e3868741fb6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.767564 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.767415    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-data-science-collector-ta-client-cert" pod "ec082f00-8899-45c3-9b20-0e3868741fb6" (UID: "ec082f00-8899-45c3-9b20-0e3868741fb6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.767564 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.767509    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-tls-certs" pod "ec082f00-8899-45c3-9b20-0e3868741fb6" (UID: "ec082f00-8899-45c3-9b20-0e3868741fb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.767755 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.767733    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-monitoring-tls-certs" pod "ec082f00-8899-45c3-9b20-0e3868741fb6" (UID: "ec082f00-8899-45c3-9b20-0e3868741fb6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:42.865573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.865497    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5m2p\" (UniqueName: \"kubernetes.io/projected/ec082f00-8899-45c3-9b20-0e3868741fb6-kube-api-access-q5m2p\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.865573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.865526    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.865573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.865540    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.865573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.865552    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec082f00-8899-45c3-9b20-0e3868741fb6-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.865573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.865561    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:42.865573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:42.865570    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec082f00-8899-45c3-9b20-0e3868741fb6-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:43.158547 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:43.158516    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" path="/var/lib/kubelet/pods/6aeb82e9-d212-419e-9cc3-24fcf9c91d33/volumes"
Oct 30 03:40:43.158943 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:43.158931    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" path="/var/lib/kubelet/pods/b01a2d62-bcab-408e-acd7-7e875276d0e0/volumes"
Oct 30 03:40:43.159361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:43.159350    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" path="/var/lib/kubelet/pods/c42d6464-fd67-4e6e-a690-7d1c1ea93cc8/volumes"
Oct 30 03:40:43.637586 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:43.637503    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec082f00-8899-45c3-9b20-0e3868741fb6","Type":"ContainerDied","Data":"13805ee7570079dfc8cbf88e0aaa16f300b317134bf09d7def34ccfcff35d503"}
Oct 30 03:40:43.637586 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:43.637530    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:40:43.638059 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:43.637609    4334 scope.go:122] "RemoveContainer" containerID="e49fbfc3d4dc36e69a780d42e61a061c75433462143f56ec24e94a75db99b3da"
Oct 30 03:40:43.691766 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:43.691738    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:40:43.697893 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:43.697871    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:40:45.158618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:45.158583    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec082f00-8899-45c3-9b20-0e3868741fb6" path="/var/lib/kubelet/pods/ec082f00-8899-45c3-9b20-0e3868741fb6/volumes"
Oct 30 03:40:47.643960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:47.643917    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.37:9093/-/ready\": dial tcp 10.133.0.37:9093: connect: connection refused"
Oct 30 03:40:48.366185 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366152    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:40:48.366635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366617    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="init-config-reloader"
Oct 30 03:40:48.366635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366635    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="init-config-reloader"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366646    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="config-reloader"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366652    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="config-reloader"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366660    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="prometheus"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366665    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="prometheus"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366682    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerName="kube-rbac-proxy"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366687    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerName="kube-rbac-proxy"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366694    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerName="prom-label-proxy"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366698    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerName="prom-label-proxy"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366706    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ec082f00-8899-45c3-9b20-0e3868741fb6" containerName="otc-container"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366711    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec082f00-8899-45c3-9b20-0e3868741fb6" containerName="otc-container"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366725    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerName="kube-rbac-proxy"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366729    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerName="kube-rbac-proxy"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366735    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerName="prom-label-proxy"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366739    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerName="prom-label-proxy"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366749    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="thanos-sidecar"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366753    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="thanos-sidecar"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366758    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ff1316ee-a718-4fbe-8842-c1916cdbb639" containerName="curl"
Oct 30 03:40:48.366782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366763    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1316ee-a718-4fbe-8842-c1916cdbb639" containerName="curl"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366814    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="config-reloader"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366823    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="prometheus"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366831    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerName="kube-rbac-proxy"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366837    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aeb82e9-d212-419e-9cc3-24fcf9c91d33" containerName="prom-label-proxy"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366842    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec082f00-8899-45c3-9b20-0e3868741fb6" containerName="otc-container"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366848    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerName="kube-rbac-proxy"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366853    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01a2d62-bcab-408e-acd7-7e875276d0e0" containerName="prom-label-proxy"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366858    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42d6464-fd67-4e6e-a690-7d1c1ea93cc8" containerName="thanos-sidecar"
Oct 30 03:40:48.367310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.366864    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1316ee-a718-4fbe-8842-c1916cdbb639" containerName="curl"
Oct 30 03:40:48.371368 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.371349    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.387933 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.387907    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:40:48.398536 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.398501    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:40:48.398536 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.398517    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:40:48.398676 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.398649    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-4tr57\""
Oct 30 03:40:48.400280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.400251    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:40:48.512723 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.512683    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9dfn\" (UniqueName: \"kubernetes.io/projected/63df2c9b-0935-4201-9b50-587308046f0c-kube-api-access-f9dfn\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.512723 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.512725    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.512926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.512866    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.512926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.512920    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/63df2c9b-0935-4201-9b50-587308046f0c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.613662 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.613632    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9dfn\" (UniqueName: \"kubernetes.io/projected/63df2c9b-0935-4201-9b50-587308046f0c-kube-api-access-f9dfn\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.613662 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.613665    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.613869 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.613703    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.613869 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.613750    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/63df2c9b-0935-4201-9b50-587308046f0c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.614460 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.614430    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/63df2c9b-0935-4201-9b50-587308046f0c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.616406 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.616350    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.616406 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.616377    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.633580 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.633550    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9dfn\" (UniqueName: \"kubernetes.io/projected/63df2c9b-0935-4201-9b50-587308046f0c-kube-api-access-f9dfn\") pod \"data-science-collector-collector-0\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.685429 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.685404    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:40:48.981944 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.981702    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:40:48.984110 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:40:48.984083    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63df2c9b_0935_4201_9b50_587308046f0c.slice/crio-95fc0b85cdd7f0e1b6ef22fa2f4b194525d7a700f97626d298b4b52fe3ff6417 WatchSource:0}: Error finding container 95fc0b85cdd7f0e1b6ef22fa2f4b194525d7a700f97626d298b4b52fe3ff6417: Status 404 returned error can't find the container with id 95fc0b85cdd7f0e1b6ef22fa2f4b194525d7a700f97626d298b4b52fe3ff6417
Oct 30 03:40:48.986050 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.986021    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:40:48.986131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:48.986083    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:40:49.301104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.301029    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:40:49.305260 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.305234    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.323263 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.323240    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:40:49.334486 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.334421    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:40:49.334605 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.334498    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:40:49.334605 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.334499    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:40:49.334605 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.334506    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:40:49.334767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.334494    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-cfzdv\""
Oct 30 03:40:49.334767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.334667    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:40:49.334898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.334859    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:40:49.334943 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.334930    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:40:49.422603 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.422575    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.422721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.422611    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.422721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.422634    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.422721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.422687    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.422834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.422736    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68wd\" (UniqueName: \"kubernetes.io/projected/dbcca83b-5e9d-4690-9120-e98330a3835b-kube-api-access-x68wd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.422834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.422784    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.422834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.422815    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.422927 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.422861    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.523946 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.523916    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.523952    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.523978    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.524010    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68wd\" (UniqueName: \"kubernetes.io/projected/dbcca83b-5e9d-4690-9120-e98330a3835b-kube-api-access-x68wd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.524043    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.524066    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.524096    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524411 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.524184    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524801 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.524752    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524900 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.524851    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.524900 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.524879    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.525060 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.525042    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.526550 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.526532    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.526952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.526932    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.527025 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.526979    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.544774 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.544752    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68wd\" (UniqueName: \"kubernetes.io/projected/dbcca83b-5e9d-4690-9120-e98330a3835b-kube-api-access-x68wd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:49.669056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.669027    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"63df2c9b-0935-4201-9b50-587308046f0c","Type":"ContainerStarted","Data":"31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd"}
Oct 30 03:40:49.669182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.669076    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"63df2c9b-0935-4201-9b50-587308046f0c","Type":"ContainerStarted","Data":"95fc0b85cdd7f0e1b6ef22fa2f4b194525d7a700f97626d298b4b52fe3ff6417"}
Oct 30 03:40:49.710834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:49.710791    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.7107777130000001 podStartE2EDuration="1.710777713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:49.709979588 +0000 UTC m=+3528.716569852" watchObservedRunningTime="2026-10-30 03:40:49.710777713 +0000 UTC m=+3528.717367966"
Oct 30 03:40:50.977947 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:50.977910    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:40:52.027724 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.027699    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:40:52.148084 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.148011    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-kube-api-access-zjbc5\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-kube-api-access-zjbc5\") pod \"f0b95646-94f7-4f0c-b193-b1a530517b68\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") "
Oct 30 03:40:52.148084 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.148056    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-alertmanager-data-science-monitoringstack-db\") pod \"f0b95646-94f7-4f0c-b193-b1a530517b68\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") "
Oct 30 03:40:52.148084 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.148084    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-config-out\") pod \"f0b95646-94f7-4f0c-b193-b1a530517b68\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") "
Oct 30 03:40:52.148346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.148105    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-cluster-tls-config\") pod \"f0b95646-94f7-4f0c-b193-b1a530517b68\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") "
Oct 30 03:40:52.148346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.148136    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-tls-assets\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-tls-assets\") pod \"f0b95646-94f7-4f0c-b193-b1a530517b68\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") "
Oct 30 03:40:52.148346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.148156    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-web-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-web-config\") pod \"f0b95646-94f7-4f0c-b193-b1a530517b68\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") "
Oct 30 03:40:52.148346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.148190    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-config-volume\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-config-volume\") pod \"f0b95646-94f7-4f0c-b193-b1a530517b68\" (UID: \"f0b95646-94f7-4f0c-b193-b1a530517b68\") "
Oct 30 03:40:52.148595 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.148561    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-alertmanager-data-science-monitoringstack-db" pod "f0b95646-94f7-4f0c-b193-b1a530517b68" (UID: "f0b95646-94f7-4f0c-b193-b1a530517b68"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:52.150522 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.150480    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-config-volume" pod "f0b95646-94f7-4f0c-b193-b1a530517b68" (UID: "f0b95646-94f7-4f0c-b193-b1a530517b68"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:52.150629 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.150584    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-kube-api-access-zjbc5" pod "f0b95646-94f7-4f0c-b193-b1a530517b68" (UID: "f0b95646-94f7-4f0c-b193-b1a530517b68"). InnerVolumeSpecName "kube-api-access-zjbc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:52.150803 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.150783    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-config-out" pod "f0b95646-94f7-4f0c-b193-b1a530517b68" (UID: "f0b95646-94f7-4f0c-b193-b1a530517b68"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:52.150864 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.150820    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-tls-assets" pod "f0b95646-94f7-4f0c-b193-b1a530517b68" (UID: "f0b95646-94f7-4f0c-b193-b1a530517b68"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:52.154287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.154266    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-cluster-tls-config" pod "f0b95646-94f7-4f0c-b193-b1a530517b68" (UID: "f0b95646-94f7-4f0c-b193-b1a530517b68"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:52.161143 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.161119    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-web-config" pod "f0b95646-94f7-4f0c-b193-b1a530517b68" (UID: "f0b95646-94f7-4f0c-b193-b1a530517b68"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:52.249502 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.249474    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.249502 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.249498    4334 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-cluster-tls-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.249638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.249508    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.249638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.249518    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.249638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.249526    4334 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f0b95646-94f7-4f0c-b193-b1a530517b68-config-volume\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.249638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.249534    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbc5\" (UniqueName: \"kubernetes.io/projected/f0b95646-94f7-4f0c-b193-b1a530517b68-kube-api-access-zjbc5\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.249638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.249545    4334 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/f0b95646-94f7-4f0c-b193-b1a530517b68-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.684951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.684919    4334 generic.go:356] "Generic (PLEG): container finished" podID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerID="3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4" exitCode=0
Oct 30 03:40:52.685146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.684995    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 03:40:52.685146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.685005    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"f0b95646-94f7-4f0c-b193-b1a530517b68","Type":"ContainerDied","Data":"3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4"}
Oct 30 03:40:52.685146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.685060    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"f0b95646-94f7-4f0c-b193-b1a530517b68","Type":"ContainerDied","Data":"eb43f5cd71b6d2bf59906531bd8733d833e829e3ec4d40138740eb2e76a58e57"}
Oct 30 03:40:52.685146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.685087    4334 scope.go:122] "RemoveContainer" containerID="33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7"
Oct 30 03:40:52.693676 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.693657    4334 scope.go:122] "RemoveContainer" containerID="3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4"
Oct 30 03:40:52.701314 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.701298    4334 scope.go:122] "RemoveContainer" containerID="3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371"
Oct 30 03:40:52.710767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.710746    4334 scope.go:122] "RemoveContainer" containerID="33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7"
Oct 30 03:40:52.711006 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:52.710987    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7\": container with ID starting with 33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7 not found: ID does not exist" containerID="33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7"
Oct 30 03:40:52.711066 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.711014    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7"} err="failed to get container status \"33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7\": rpc error: code = NotFound desc = could not find container \"33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7\": container with ID starting with 33a19fd0f6751304131aa5f1e4b0e35fc6ed7cb51bb197117646b4c89185a8b7 not found: ID does not exist"
Oct 30 03:40:52.711066 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.711029    4334 scope.go:122] "RemoveContainer" containerID="3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4"
Oct 30 03:40:52.711303 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:52.711286    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4\": container with ID starting with 3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4 not found: ID does not exist" containerID="3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4"
Oct 30 03:40:52.711361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.711315    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4"} err="failed to get container status \"3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4\": rpc error: code = NotFound desc = could not find container \"3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4\": container with ID starting with 3bc8f24f8ec96f176c46e5e41b22d292d95023961f818f3c11e3ae8e77233db4 not found: ID does not exist"
Oct 30 03:40:52.711361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.711338    4334 scope.go:122] "RemoveContainer" containerID="3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371"
Oct 30 03:40:52.711663 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:40:52.711646    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371\": container with ID starting with 3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371 not found: ID does not exist" containerID="3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371"
Oct 30 03:40:52.711721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.711667    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371"} err="failed to get container status \"3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371\": rpc error: code = NotFound desc = could not find container \"3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371\": container with ID starting with 3e798a87b540224be78856ed38185b3fa88dd472c37515ec8f5dd85e18bfa371 not found: ID does not exist"
Oct 30 03:40:52.741058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.741036    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:40:52.754186 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:52.754166    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 03:40:53.159912 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:53.159884    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" path="/var/lib/kubelet/pods/f0b95646-94f7-4f0c-b193-b1a530517b68/volumes"
Oct 30 03:40:53.862564 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:53.862529    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:53.892428 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:53.892398    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:53.963807 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:53.963777    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:53.963944 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:53.963894    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:53.979551 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:53.979519    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:54.080263 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.080237    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:54.107098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.107073    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:54.125214 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.125151    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:54.450674 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.450650    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:40:54.452673 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:40:54.452644    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcca83b_5e9d_4690_9120_e98330a3835b.slice/crio-c2feffdd74346f0345f0b3ee2060386bab2dd609cd073e893206d8b7a4462fd4 WatchSource:0}: Error finding container c2feffdd74346f0345f0b3ee2060386bab2dd609cd073e893206d8b7a4462fd4: Status 404 returned error can't find the container with id c2feffdd74346f0345f0b3ee2060386bab2dd609cd073e893206d8b7a4462fd4
Oct 30 03:40:54.698981 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.698943    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbcca83b-5e9d-4690-9120-e98330a3835b","Type":"ContainerStarted","Data":"b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1"}
Oct 30 03:40:54.699121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.699000    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbcca83b-5e9d-4690-9120-e98330a3835b","Type":"ContainerStarted","Data":"e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420"}
Oct 30 03:40:54.699121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.699023    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbcca83b-5e9d-4690-9120-e98330a3835b","Type":"ContainerStarted","Data":"84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04"}
Oct 30 03:40:54.699121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.699046    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbcca83b-5e9d-4690-9120-e98330a3835b","Type":"ContainerStarted","Data":"c2feffdd74346f0345f0b3ee2060386bab2dd609cd073e893206d8b7a4462fd4"}
Oct 30 03:40:54.699121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.699073    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:54.701482 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.701420    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:54.756071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:54.756031    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=5.756018629 podStartE2EDuration="5.756018629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:54.742263126 +0000 UTC m=+3533.748853381" watchObservedRunningTime="2026-10-30 03:40:54.756018629 +0000 UTC m=+3533.762608882"
Oct 30 03:40:55.703704 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:55.703661    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:55.704077 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:55.703726    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:40:55.711172 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:40:55.711148    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:17.714719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:17.714622    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:41:20.978271 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:20.978238    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:41:27.714707 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:27.714649    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:41:37.714091 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:37.714054    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:39.898033 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:39.897998    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:41:39.898437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:39.898195    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="63df2c9b-0935-4201-9b50-587308046f0c" containerName="otc-container" containerID="cri-o://31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd" gracePeriod=30
Oct 30 03:41:40.151170 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.151107    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:40.233565 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.233535    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63df2c9b-0935-4201-9b50-587308046f0c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/63df2c9b-0935-4201-9b50-587308046f0c-otc-internal\") pod \"63df2c9b-0935-4201-9b50-587308046f0c\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") "
Oct 30 03:41:40.233740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.233593    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-otlp-tls-certs\") pod \"63df2c9b-0935-4201-9b50-587308046f0c\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") "
Oct 30 03:41:40.233740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.233666    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-monitoring-tls-certs\") pod \"63df2c9b-0935-4201-9b50-587308046f0c\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") "
Oct 30 03:41:40.233740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.233730    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63df2c9b-0935-4201-9b50-587308046f0c-kube-api-access-f9dfn\" (UniqueName: \"kubernetes.io/projected/63df2c9b-0935-4201-9b50-587308046f0c-kube-api-access-f9dfn\") pod \"63df2c9b-0935-4201-9b50-587308046f0c\" (UID: \"63df2c9b-0935-4201-9b50-587308046f0c\") "
Oct 30 03:41:40.233945 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.233915    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63df2c9b-0935-4201-9b50-587308046f0c-otc-internal" pod "63df2c9b-0935-4201-9b50-587308046f0c" (UID: "63df2c9b-0935-4201-9b50-587308046f0c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:40.234023 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.234004    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/63df2c9b-0935-4201-9b50-587308046f0c-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:40.235797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.235760    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-monitoring-tls-certs" pod "63df2c9b-0935-4201-9b50-587308046f0c" (UID: "63df2c9b-0935-4201-9b50-587308046f0c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:40.235797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.235775    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-otlp-tls-certs" pod "63df2c9b-0935-4201-9b50-587308046f0c" (UID: "63df2c9b-0935-4201-9b50-587308046f0c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:40.236130 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.236110    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63df2c9b-0935-4201-9b50-587308046f0c-kube-api-access-f9dfn" pod "63df2c9b-0935-4201-9b50-587308046f0c" (UID: "63df2c9b-0935-4201-9b50-587308046f0c"). InnerVolumeSpecName "kube-api-access-f9dfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:40.334546 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.334517    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:40.334546 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.334541    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/63df2c9b-0935-4201-9b50-587308046f0c-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:40.334732 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.334555    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9dfn\" (UniqueName: \"kubernetes.io/projected/63df2c9b-0935-4201-9b50-587308046f0c-kube-api-access-f9dfn\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:40.382111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.382082    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:40.382336 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.382307    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway" containerID="cri-o://e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420" gracePeriod=30
Oct 30 03:41:40.382547 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.382329    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway-opa" containerID="cri-o://b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1" gracePeriod=30
Oct 30 03:41:40.382547 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.382285    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo" containerID="cri-o://84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04" gracePeriod=30
Oct 30 03:41:40.535904 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:40.535870    4334 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcca83b_5e9d_4690_9120_e98330a3835b.slice/crio-conmon-84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:41:40.708930 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.708893    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:41:40.911178 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.911142    4334 generic.go:356] "Generic (PLEG): container finished" podID="63df2c9b-0935-4201-9b50-587308046f0c" containerID="31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd" exitCode=0
Oct 30 03:41:40.911689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.911215    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:40.911689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.911234    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"63df2c9b-0935-4201-9b50-587308046f0c","Type":"ContainerDied","Data":"31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd"}
Oct 30 03:41:40.911689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.911331    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"63df2c9b-0935-4201-9b50-587308046f0c","Type":"ContainerDied","Data":"95fc0b85cdd7f0e1b6ef22fa2f4b194525d7a700f97626d298b4b52fe3ff6417"}
Oct 30 03:41:40.911689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.911366    4334 scope.go:122] "RemoveContainer" containerID="31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd"
Oct 30 03:41:40.913902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.913881    4334 generic.go:356] "Generic (PLEG): container finished" podID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerID="b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1" exitCode=0
Oct 30 03:41:40.913902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.913902    4334 generic.go:356] "Generic (PLEG): container finished" podID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerID="84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04" exitCode=0
Oct 30 03:41:40.914069 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.913947    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbcca83b-5e9d-4690-9120-e98330a3835b","Type":"ContainerDied","Data":"b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1"}
Oct 30 03:41:40.914069 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.913989    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbcca83b-5e9d-4690-9120-e98330a3835b","Type":"ContainerDied","Data":"84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04"}
Oct 30 03:41:40.923614 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.923598    4334 scope.go:122] "RemoveContainer" containerID="31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd"
Oct 30 03:41:40.923890 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:40.923872    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd\": container with ID starting with 31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd not found: ID does not exist" containerID="31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd"
Oct 30 03:41:40.923927 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.923899    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd"} err="failed to get container status \"31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd\": rpc error: code = NotFound desc = could not find container \"31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd\": container with ID starting with 31802745e4285877159b2b85ddf6b7155dc0ee14f2430b07e6dbd03f8a8c2dbd not found: ID does not exist"
Oct 30 03:41:40.971108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.971082    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:41:40.974687 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:40.974668    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:41:41.159356 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.159321    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63df2c9b-0935-4201-9b50-587308046f0c" path="/var/lib/kubelet/pods/63df2c9b-0935-4201-9b50-587308046f0c/volumes"
Oct 30 03:41:41.630116 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.630092    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:41.744769 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.744696    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-rbac\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.744769 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.744750    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbcca83b-5e9d-4690-9120-e98330a3835b-kube-api-access-x68wd\" (UniqueName: \"kubernetes.io/projected/dbcca83b-5e9d-4690-9120-e98330a3835b-kube-api-access-x68wd\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.744979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.744795    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.744979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.744818    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-tenants\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.744979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.744866    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.744979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.744899    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-conf\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.744979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.744932    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-mtls\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.745241 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.745015    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.745241 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.745048    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"dbcca83b-5e9d-4690-9120-e98330a3835b\" (UID: \"dbcca83b-5e9d-4690-9120-e98330a3835b\") "
Oct 30 03:41:41.745241 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.745130    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-rbac" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:41.745390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.745309    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:41.745390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.745366    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-serving-cabundle" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:41.745707 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.745676    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-conf" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:41.745822 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.745676    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-ca-bundle" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:41.747310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.747285    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcca83b-5e9d-4690-9120-e98330a3835b-kube-api-access-x68wd" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "kube-api-access-x68wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:41.747310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.747282    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-tenants" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:41.747516 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.747306    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:41.747566 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.747520    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-mtls" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:41.748228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.748208    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "dbcca83b-5e9d-4690-9120-e98330a3835b" (UID: "dbcca83b-5e9d-4690-9120-e98330a3835b"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:41:41.846486 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.846441    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:41.846618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.846504    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:41:41.846618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.846523    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:41.846618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.846537    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68wd\" (UniqueName: \"kubernetes.io/projected/dbcca83b-5e9d-4690-9120-e98330a3835b-kube-api-access-x68wd\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:41.846618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.846551    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:41.846618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.846565    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbcca83b-5e9d-4690-9120-e98330a3835b-gateway-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:41.846618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.846579    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:41.846618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.846593    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbcca83b-5e9d-4690-9120-e98330a3835b-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:41.854405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.854390    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:41:41.921761 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.921736    4334 generic.go:356] "Generic (PLEG): container finished" podID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerID="e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420" exitCode=0
Oct 30 03:41:41.922151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.921807    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbcca83b-5e9d-4690-9120-e98330a3835b","Type":"ContainerDied","Data":"e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420"}
Oct 30 03:41:41.922151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.921828    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:41.922151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.921855    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbcca83b-5e9d-4690-9120-e98330a3835b","Type":"ContainerDied","Data":"c2feffdd74346f0345f0b3ee2060386bab2dd609cd073e893206d8b7a4462fd4"}
Oct 30 03:41:41.922151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.921879    4334 scope.go:122] "RemoveContainer" containerID="b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1"
Oct 30 03:41:41.930240 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.929960    4334 scope.go:122] "RemoveContainer" containerID="e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420"
Oct 30 03:41:41.937380 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.937364    4334 scope.go:122] "RemoveContainer" containerID="84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04"
Oct 30 03:41:41.944580 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.944561    4334 scope.go:122] "RemoveContainer" containerID="b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1"
Oct 30 03:41:41.944819 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:41.944800    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1\": container with ID starting with b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1 not found: ID does not exist" containerID="b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1"
Oct 30 03:41:41.944860 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.944826    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1"} err="failed to get container status \"b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1\": rpc error: code = NotFound desc = could not find container \"b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1\": container with ID starting with b65536ca82eb66d7c907b6fbeee0527831abdd56b8045a6ea4a3918101df18d1 not found: ID does not exist"
Oct 30 03:41:41.944860 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.944841    4334 scope.go:122] "RemoveContainer" containerID="e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420"
Oct 30 03:41:41.945060 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:41.945044    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420\": container with ID starting with e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420 not found: ID does not exist" containerID="e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420"
Oct 30 03:41:41.945097 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.945069    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420"} err="failed to get container status \"e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420\": rpc error: code = NotFound desc = could not find container \"e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420\": container with ID starting with e6baf0728ceb108deacfbd4d8931e968a49f78d5bb95e6ac4114606a7c598420 not found: ID does not exist"
Oct 30 03:41:41.945097 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.945083    4334 scope.go:122] "RemoveContainer" containerID="84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04"
Oct 30 03:41:41.945279 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:41.945257    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04\": container with ID starting with 84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04 not found: ID does not exist" containerID="84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04"
Oct 30 03:41:41.945315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.945283    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04"} err="failed to get container status \"84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04\": rpc error: code = NotFound desc = could not find container \"84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04\": container with ID starting with 84e1448d329454fd812bf75e20b5a6d3aea60f6a8d0e607461863796051a2e04 not found: ID does not exist"
Oct 30 03:41:41.947643 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.947625    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\""
Oct 30 03:41:41.977089 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.977067    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:41.990401 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:41.990380    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:43.158796 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:43.158756    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" path="/var/lib/kubelet/pods/dbcca83b-5e9d-4690-9120-e98330a3835b/volumes"
Oct 30 03:41:44.180711 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.180682    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:41:44.181181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181088    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="init-config-reloader"
Oct 30 03:41:44.181181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181105    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="init-config-reloader"
Oct 30 03:41:44.181181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181117    4334 cpu_manager.go:414] "containerMap: removing container" podUID="63df2c9b-0935-4201-9b50-587308046f0c" containerName="otc-container"
Oct 30 03:41:44.181181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181124    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63df2c9b-0935-4201-9b50-587308046f0c" containerName="otc-container"
Oct 30 03:41:44.181181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181144    4334 cpu_manager.go:414] "containerMap: removing container" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway"
Oct 30 03:41:44.181181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181152    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway"
Oct 30 03:41:44.181181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181161    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="alertmanager"
Oct 30 03:41:44.181181 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181168    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="alertmanager"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181196    4334 cpu_manager.go:414] "containerMap: removing container" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181203    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181218    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="config-reloader"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181225    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="config-reloader"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181236    4334 cpu_manager.go:414] "containerMap: removing container" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway-opa"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181242    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway-opa"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181325    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181336    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="alertmanager"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181348    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181357    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b95646-94f7-4f0c-b193-b1a530517b68" containerName="config-reloader"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181365    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbcca83b-5e9d-4690-9120-e98330a3835b" containerName="tempo-gateway-opa"
Oct 30 03:41:44.181620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.181375    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="63df2c9b-0935-4201-9b50-587308046f0c" containerName="otc-container"
Oct 30 03:41:44.186579 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.186546    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.203146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.203109    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:41:44.205949 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.205927    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:41:44.216712 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.216659    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:41:44.216712 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.216661    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:41:44.216918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.216743    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:41:44.263270 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.262585    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dts7r\" (UniqueName: \"kubernetes.io/projected/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-kube-api-access-dts7r\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.263270 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.262723    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.263270 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.262764    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.263270 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.262799    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.303961 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.303937    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:41:44.363334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.363308    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.363514 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.363339    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.363514 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.363367    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dts7r\" (UniqueName: \"kubernetes.io/projected/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-kube-api-access-dts7r\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.363514 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.363419    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.363514 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:44.363467    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found
Oct 30 03:41:44.363738 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:44.363533    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs podName:cc91fe2b-abb8-473c-8b87-2a931dcd06cd nodeName:}" failed. No retries permitted until 2026-10-30 03:41:44.863516062 +0000 UTC m=+3583.870106294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "cc91fe2b-abb8-473c-8b87-2a931dcd06cd") : secret "data-science-collector-collector-headless-tls" not found
Oct 30 03:41:44.364031 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.364013    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.365912 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.365883    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.387719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.387691    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dts7r\" (UniqueName: \"kubernetes.io/projected/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-kube-api-access-dts7r\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.867140 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.867103    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:44.869618 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:44.869593    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:45.099537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.099506    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:45.397308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.397247    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:41:45.400167 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:41:45.400137    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc91fe2b_abb8_473c_8b87_2a931dcd06cd.slice/crio-da614cd7e2a67f7586b5a7c48965a1d80def117c143ab2226b82f54e02169b68 WatchSource:0}: Error finding container da614cd7e2a67f7586b5a7c48965a1d80def117c143ab2226b82f54e02169b68: Status 404 returned error can't find the container with id da614cd7e2a67f7586b5a7c48965a1d80def117c143ab2226b82f54e02169b68
Oct 30 03:41:45.402306 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.402273    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:41:45.402361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.402333    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:41:45.739027 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.738995    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:45.742573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.742552    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.759571 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.759544    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-qf54q\""
Oct 30 03:41:45.772629 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.772601    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:41:45.772741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.772630    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:41:45.772741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.772730    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:41:45.772854 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.772768    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:41:45.772917 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.772858    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:41:45.772917 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.772871    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:41:45.773024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.773011    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:41:45.774539 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.774516    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:45.872150 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.872118    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.872315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.872157    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.872315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.872180    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.872315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.872255    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.872315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.872300    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.872554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.872411    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.872554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.872510    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.872554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.872535    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf7g2\" (UniqueName: \"kubernetes.io/projected/dbccc4c5-416d-4308-814f-712f14d310c3-kube-api-access-bf7g2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.942806 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.942777    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cc91fe2b-abb8-473c-8b87-2a931dcd06cd","Type":"ContainerStarted","Data":"f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243"}
Oct 30 03:41:45.942947 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.942834    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cc91fe2b-abb8-473c-8b87-2a931dcd06cd","Type":"ContainerStarted","Data":"da614cd7e2a67f7586b5a7c48965a1d80def117c143ab2226b82f54e02169b68"}
Oct 30 03:41:45.973852 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.973828    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.973996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.973868    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.973996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.973891    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.973996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.973928    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.973996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.973962    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.974229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.974061    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.974229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.974093    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.974229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.974136    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf7g2\" (UniqueName: \"kubernetes.io/projected/dbccc4c5-416d-4308-814f-712f14d310c3-kube-api-access-bf7g2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.974655 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.974636    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.975058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.975034    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.975232 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.975200    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.975302 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.975278    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.976651 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.976619    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.976651 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.976639    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.976770 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.976681    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:45.989784 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:45.989718    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.989706258 podStartE2EDuration="1.989706258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:45.989407922 +0000 UTC m=+3584.995998174" watchObservedRunningTime="2026-10-30 03:41:45.989706258 +0000 UTC m=+3584.996296512"
Oct 30 03:41:46.000164 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:46.000136    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf7g2\" (UniqueName: \"kubernetes.io/projected/dbccc4c5-416d-4308-814f-712f14d310c3-kube-api-access-bf7g2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:50.978092 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:41:50.978059    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:41:54.642503 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:54.642466    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:54.673410 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:54.673379    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:54.743738 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:54.743701    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:54.743852 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:54.743831    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:54.759128 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:54.759103    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:54.862111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:54.862080    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:54.889264 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:54.889243    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:55.060654 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.060624    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:55.388669 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:41:55.388631    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbccc4c5_416d_4308_814f_712f14d310c3.slice/crio-0eb3bac5d63588d8deaca677132dcf945ff511757907fbe9123ff09c7fa8e478 WatchSource:0}: Error finding container 0eb3bac5d63588d8deaca677132dcf945ff511757907fbe9123ff09c7fa8e478: Status 404 returned error can't find the container with id 0eb3bac5d63588d8deaca677132dcf945ff511757907fbe9123ff09c7fa8e478
Oct 30 03:41:55.400102 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.400070    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:55.990914 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.990880    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbccc4c5-416d-4308-814f-712f14d310c3","Type":"ContainerStarted","Data":"ca900a1269a0025adecc6117a5fec28877a89bb34dbe468fe3aa662d2f684cef"}
Oct 30 03:41:55.991361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.990942    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbccc4c5-416d-4308-814f-712f14d310c3","Type":"ContainerStarted","Data":"db8ca406e6462f97b3d6380f9c6dd8d04a1772933ab5f4ebd7c12708fa099e9c"}
Oct 30 03:41:55.991361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.990966    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbccc4c5-416d-4308-814f-712f14d310c3","Type":"ContainerStarted","Data":"af0348f23e6ffc8987948124157ed81fdc8ab18b45c9bee82693d16085f8d4eb"}
Oct 30 03:41:55.991361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.990990    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbccc4c5-416d-4308-814f-712f14d310c3","Type":"ContainerStarted","Data":"0eb3bac5d63588d8deaca677132dcf945ff511757907fbe9123ff09c7fa8e478"}
Oct 30 03:41:55.991361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.991090    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:55.991361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.991121    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:55.991361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.991144    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:55.993532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.993507    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:55.999543 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:55.999523    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:56.066204 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:56.066153    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=11.066139227 podStartE2EDuration="11.066139227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:56.052245909 +0000 UTC m=+3595.058836163" watchObservedRunningTime="2026-10-30 03:41:56.066139227 +0000 UTC m=+3595.072729483"
Oct 30 03:41:59.368043 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.368008    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:41:59.368834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.368799    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="cc91fe2b-abb8-473c-8b87-2a931dcd06cd" containerName="otc-container" containerID="cri-o://f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243" gracePeriod=30
Oct 30 03:41:59.617396 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.617370    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:41:59.687164 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.687135    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs\") pod \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") "
Oct 30 03:41:59.687366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.687195    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-kube-api-access-dts7r\" (UniqueName: \"kubernetes.io/projected/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-kube-api-access-dts7r\") pod \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") "
Oct 30 03:41:59.687366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.687240    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otc-internal\") pod \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") "
Oct 30 03:41:59.687366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.687267    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-monitoring-tls-certs\") pod \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\" (UID: \"cc91fe2b-abb8-473c-8b87-2a931dcd06cd\") "
Oct 30 03:41:59.687637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.687609    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otc-internal" pod "cc91fe2b-abb8-473c-8b87-2a931dcd06cd" (UID: "cc91fe2b-abb8-473c-8b87-2a931dcd06cd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:59.689382 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.689357    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-monitoring-tls-certs" pod "cc91fe2b-abb8-473c-8b87-2a931dcd06cd" (UID: "cc91fe2b-abb8-473c-8b87-2a931dcd06cd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:59.689805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.689784    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs" pod "cc91fe2b-abb8-473c-8b87-2a931dcd06cd" (UID: "cc91fe2b-abb8-473c-8b87-2a931dcd06cd"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:59.689883 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.689787    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-kube-api-access-dts7r" pod "cc91fe2b-abb8-473c-8b87-2a931dcd06cd" (UID: "cc91fe2b-abb8-473c-8b87-2a931dcd06cd"). InnerVolumeSpecName "kube-api-access-dts7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:59.787959 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.787930    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.787959 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.787956    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dts7r\" (UniqueName: \"kubernetes.io/projected/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-kube-api-access-dts7r\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.788114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.787966    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.788114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.787975    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc91fe2b-abb8-473c-8b87-2a931dcd06cd-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:41:59.842775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.842747    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:59.842986 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.842954    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo" containerID="cri-o://af0348f23e6ffc8987948124157ed81fdc8ab18b45c9bee82693d16085f8d4eb" gracePeriod=30
Oct 30 03:41:59.843113 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.843020    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway" containerID="cri-o://db8ca406e6462f97b3d6380f9c6dd8d04a1772933ab5f4ebd7c12708fa099e9c" gracePeriod=30
Oct 30 03:41:59.843113 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:41:59.843029    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway-opa" containerID="cri-o://ca900a1269a0025adecc6117a5fec28877a89bb34dbe468fe3aa662d2f684cef" gracePeriod=30
Oct 30 03:42:00.012131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.012099    4334 generic.go:356] "Generic (PLEG): container finished" podID="dbccc4c5-416d-4308-814f-712f14d310c3" containerID="ca900a1269a0025adecc6117a5fec28877a89bb34dbe468fe3aa662d2f684cef" exitCode=0
Oct 30 03:42:00.012290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.012167    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbccc4c5-416d-4308-814f-712f14d310c3","Type":"ContainerDied","Data":"ca900a1269a0025adecc6117a5fec28877a89bb34dbe468fe3aa662d2f684cef"}
Oct 30 03:42:00.013408 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.013388    4334 generic.go:356] "Generic (PLEG): container finished" podID="cc91fe2b-abb8-473c-8b87-2a931dcd06cd" containerID="f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243" exitCode=0
Oct 30 03:42:00.013547 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.013417    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cc91fe2b-abb8-473c-8b87-2a931dcd06cd","Type":"ContainerDied","Data":"f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243"}
Oct 30 03:42:00.013547 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.013442    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:00.013547 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.013501    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cc91fe2b-abb8-473c-8b87-2a931dcd06cd","Type":"ContainerDied","Data":"da614cd7e2a67f7586b5a7c48965a1d80def117c143ab2226b82f54e02169b68"}
Oct 30 03:42:00.013709 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.013547    4334 scope.go:122] "RemoveContainer" containerID="f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243"
Oct 30 03:42:00.022637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.022616    4334 scope.go:122] "RemoveContainer" containerID="f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243"
Oct 30 03:42:00.022904 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:42:00.022882    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243\": container with ID starting with f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243 not found: ID does not exist" containerID="f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243"
Oct 30 03:42:00.022960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.022916    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243"} err="failed to get container status \"f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243\": rpc error: code = NotFound desc = could not find container \"f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243\": container with ID starting with f3553dc33ea94d23c215f6655a1f03051359fa0ffdb5201e783e98ca7fa35243 not found: ID does not exist"
Oct 30 03:42:00.065871 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.065842    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:42:00.069059 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.069038    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:42:00.992067 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.992027    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.133.0.65:8081/ready\": dial tcp 10.133.0.65:8081: connect: connection refused"
Oct 30 03:42:00.992412 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:00.992027    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.65:8083/ready\": dial tcp 10.133.0.65:8083: connect: connection refused"
Oct 30 03:42:01.021841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.021814    4334 generic.go:356] "Generic (PLEG): container finished" podID="dbccc4c5-416d-4308-814f-712f14d310c3" containerID="db8ca406e6462f97b3d6380f9c6dd8d04a1772933ab5f4ebd7c12708fa099e9c" exitCode=0
Oct 30 03:42:01.021841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.021838    4334 generic.go:356] "Generic (PLEG): container finished" podID="dbccc4c5-416d-4308-814f-712f14d310c3" containerID="af0348f23e6ffc8987948124157ed81fdc8ab18b45c9bee82693d16085f8d4eb" exitCode=0
Oct 30 03:42:01.021995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.021880    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbccc4c5-416d-4308-814f-712f14d310c3","Type":"ContainerDied","Data":"db8ca406e6462f97b3d6380f9c6dd8d04a1772933ab5f4ebd7c12708fa099e9c"}
Oct 30 03:42:01.021995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.021925    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbccc4c5-416d-4308-814f-712f14d310c3","Type":"ContainerDied","Data":"af0348f23e6ffc8987948124157ed81fdc8ab18b45c9bee82693d16085f8d4eb"}
Oct 30 03:42:01.085385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.085364    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:01.159782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.159751    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc91fe2b-abb8-473c-8b87-2a931dcd06cd" path="/var/lib/kubelet/pods/cc91fe2b-abb8-473c-8b87-2a931dcd06cd/volumes"
Oct 30 03:42:01.197073 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197008    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-conf\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197073 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197051    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197254 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197079    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-ca-bundle\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197254 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197101    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbccc4c5-416d-4308-814f-712f14d310c3-kube-api-access-bf7g2\" (UniqueName: \"kubernetes.io/projected/dbccc4c5-416d-4308-814f-712f14d310c3-kube-api-access-bf7g2\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197254 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197135    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-tenants\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197254 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197178    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-rbac\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197482 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197369    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197482 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197411    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-conf" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:01.197482 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197434    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197643 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197478    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-ca-bundle" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:01.197643 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197517    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-serving-cabundle" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:01.197643 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197527    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-mtls\") pod \"dbccc4c5-416d-4308-814f-712f14d310c3\" (UID: \"dbccc4c5-416d-4308-814f-712f14d310c3\") "
Oct 30 03:42:01.197643 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197589    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-rbac" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:01.197842 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197782    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.197842 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197801    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.197842 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197819    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.197842 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.197833    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.199870 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.199848    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbccc4c5-416d-4308-814f-712f14d310c3-kube-api-access-bf7g2" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "kube-api-access-bf7g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:01.200008 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.199949    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:01.200008 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.199974    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-tenants" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:01.200113 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.200049    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-mtls" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:01.200202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.200173    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "dbccc4c5-416d-4308-814f-712f14d310c3" (UID: "dbccc4c5-416d-4308-814f-712f14d310c3"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:42:01.298971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.298938    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.298971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.298972    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf7g2\" (UniqueName: \"kubernetes.io/projected/dbccc4c5-416d-4308-814f-712f14d310c3-kube-api-access-bf7g2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.299170 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.299103    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-gateway-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.299170 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.299146    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:42:01.299170 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.299164    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbccc4c5-416d-4308-814f-712f14d310c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.308754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.308734    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:42:01.400053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:01.400009    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\""
Oct 30 03:42:02.027943 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:02.027916    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:02.028400 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:02.027912    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"dbccc4c5-416d-4308-814f-712f14d310c3","Type":"ContainerDied","Data":"0eb3bac5d63588d8deaca677132dcf945ff511757907fbe9123ff09c7fa8e478"}
Oct 30 03:42:02.028400 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:02.028096    4334 scope.go:122] "RemoveContainer" containerID="ca900a1269a0025adecc6117a5fec28877a89bb34dbe468fe3aa662d2f684cef"
Oct 30 03:42:02.036199 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:02.036180    4334 scope.go:122] "RemoveContainer" containerID="db8ca406e6462f97b3d6380f9c6dd8d04a1772933ab5f4ebd7c12708fa099e9c"
Oct 30 03:42:02.043497 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:02.043478    4334 scope.go:122] "RemoveContainer" containerID="af0348f23e6ffc8987948124157ed81fdc8ab18b45c9bee82693d16085f8d4eb"
Oct 30 03:42:02.098975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:02.098950    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:42:02.112119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:02.112098    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:42:03.158748 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:03.158719    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" path="/var/lib/kubelet/pods/dbccc4c5-416d-4308-814f-712f14d310c3/volumes"
Oct 30 03:42:04.259080 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259049    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:42:04.259495 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259479    4334 cpu_manager.go:414] "containerMap: removing container" podUID="cc91fe2b-abb8-473c-8b87-2a931dcd06cd" containerName="otc-container"
Oct 30 03:42:04.259545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259498    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc91fe2b-abb8-473c-8b87-2a931dcd06cd" containerName="otc-container"
Oct 30 03:42:04.259545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259511    4334 cpu_manager.go:414] "containerMap: removing container" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway-opa"
Oct 30 03:42:04.259545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259520    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway-opa"
Oct 30 03:42:04.259650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259577    4334 cpu_manager.go:414] "containerMap: removing container" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo"
Oct 30 03:42:04.259650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259586    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo"
Oct 30 03:42:04.259650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259604    4334 cpu_manager.go:414] "containerMap: removing container" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway"
Oct 30 03:42:04.259650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259612    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway"
Oct 30 03:42:04.259779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259683    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo"
Oct 30 03:42:04.259779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259696    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway"
Oct 30 03:42:04.259779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259707    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc91fe2b-abb8-473c-8b87-2a931dcd06cd" containerName="otc-container"
Oct 30 03:42:04.259779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.259715    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbccc4c5-416d-4308-814f-712f14d310c3" containerName="tempo-gateway-opa"
Oct 30 03:42:04.264681 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.264655    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.293865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.293817    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:42:04.294073 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.293966    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:42:04.294073 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.293996    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:42:04.294073 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.294058    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-mxwv2\""
Oct 30 03:42:04.295591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.295569    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:42:04.297979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.297957    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:42:04.297979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.297973    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:42:04.422946 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.422919    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.423107 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.422953    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.423107 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.423022    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2fn\" (UniqueName: \"kubernetes.io/projected/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-kube-api-access-8l2fn\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.423107 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.423056    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.524188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.524098    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2fn\" (UniqueName: \"kubernetes.io/projected/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-kube-api-access-8l2fn\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.524188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.524156    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.524188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.524186    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.524472 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.524207    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.524873 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.524851    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.526669 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.526644    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.526764 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.526701    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.544238 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.544213    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2fn\" (UniqueName: \"kubernetes.io/projected/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-kube-api-access-8l2fn\") pod \"data-science-collector-collector-0\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.578226 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.578201    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:42:04.877201 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.877161    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:42:04.879404 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:42:04.879377    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a20a45d_98d9_4ee8_8c7f_c418f130ca22.slice/crio-cab878b328d32c7068d94f54c87517e0a6e03d7ed10983bd6f7ff115887d40f4 WatchSource:0}: Error finding container cab878b328d32c7068d94f54c87517e0a6e03d7ed10983bd6f7ff115887d40f4: Status 404 returned error can't find the container with id cab878b328d32c7068d94f54c87517e0a6e03d7ed10983bd6f7ff115887d40f4
Oct 30 03:42:04.881665 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.881634    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:42:04.881735 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:04.881693    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:42:05.044737 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.044705    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4a20a45d-98d9-4ee8-8c7f-c418f130ca22","Type":"ContainerStarted","Data":"32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37"}
Oct 30 03:42:05.044902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.044764    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4a20a45d-98d9-4ee8-8c7f-c418f130ca22","Type":"ContainerStarted","Data":"cab878b328d32c7068d94f54c87517e0a6e03d7ed10983bd6f7ff115887d40f4"}
Oct 30 03:42:05.092556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.092497    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.092479818 podStartE2EDuration="1.092479818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:05.091342027 +0000 UTC m=+3604.097932282" watchObservedRunningTime="2026-10-30 03:42:05.092479818 +0000 UTC m=+3604.099070072"
Oct 30 03:42:05.119957 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.119925    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:42:05.123576 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.123552    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.139954 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.139925    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:42:05.140105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.139955    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:42:05.140105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.139996    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:42:05.140217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.140097    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:42:05.140217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.140117    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:42:05.140217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.140154    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:42:05.140217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.140211    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:42:05.140217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.140215    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-48c2r\""
Oct 30 03:42:05.146620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.146596    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:42:05.235982 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.235946    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.236153 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.236034    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.236153 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.236082    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.236153 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.236118    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.236291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.236199    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.236291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.236230    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnpl\" (UniqueName: \"kubernetes.io/projected/4e234bc2-d113-4b8e-b0a0-da497935ea20-kube-api-access-9lnpl\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.236369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.236301    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.236369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.236339    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337277 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.337245    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337277 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.337280    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.337304    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.337323    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.337342    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.337369    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.337393    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnpl\" (UniqueName: \"kubernetes.io/projected/4e234bc2-d113-4b8e-b0a0-da497935ea20-kube-api-access-9lnpl\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337802 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:42:05.337422    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:42:05.337802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.337436    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.337802 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:42:05.337529    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert podName:4e234bc2-d113-4b8e-b0a0-da497935ea20 nodeName:}" failed. No retries permitted until 2026-10-30 03:42:05.837508565 +0000 UTC m=+3604.844098797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:42:05.338325 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.338128    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.338325 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.338199    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.338325 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.338256    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.338491 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.338365    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.339862 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.339838    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.339862 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.339848    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.358776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.358752    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnpl\" (UniqueName: \"kubernetes.io/projected/4e234bc2-d113-4b8e-b0a0-da497935ea20-kube-api-access-9lnpl\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.841841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.841753    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:05.844537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:05.844497    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:14.815065 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:14.815028    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:14.845909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:14.845873    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:14.916259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:14.916232    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:14.916378 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:14.916344    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:14.932063 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:14.932034    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:15.035128 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:15.035092    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:15.062294 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:15.062270    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:15.349870 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:15.349837    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:15.676242 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:42:15.676208    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e234bc2_d113_4b8e_b0a0_da497935ea20.slice/crio-c35e24b50eddcf08f56b7275e08203efa33bad56d9ea5ad42b63355d7f18965f WatchSource:0}: Error finding container c35e24b50eddcf08f56b7275e08203efa33bad56d9ea5ad42b63355d7f18965f: Status 404 returned error can't find the container with id c35e24b50eddcf08f56b7275e08203efa33bad56d9ea5ad42b63355d7f18965f
Oct 30 03:42:15.687936 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:15.687913    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:42:16.097047 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:16.096960    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e234bc2-d113-4b8e-b0a0-da497935ea20","Type":"ContainerStarted","Data":"89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604"}
Oct 30 03:42:16.097047 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:16.097018    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e234bc2-d113-4b8e-b0a0-da497935ea20","Type":"ContainerStarted","Data":"9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca"}
Oct 30 03:42:16.097047 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:16.097037    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e234bc2-d113-4b8e-b0a0-da497935ea20","Type":"ContainerStarted","Data":"02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b"}
Oct 30 03:42:16.097492 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:16.097059    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e234bc2-d113-4b8e-b0a0-da497935ea20","Type":"ContainerStarted","Data":"c35e24b50eddcf08f56b7275e08203efa33bad56d9ea5ad42b63355d7f18965f"}
Oct 30 03:42:16.097492 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:16.097087    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:16.099735 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:16.099711    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:16.168013 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:16.167968    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=11.167954993 podStartE2EDuration="11.167954993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:16.167792442 +0000 UTC m=+3615.174382696" watchObservedRunningTime="2026-10-30 03:42:16.167954993 +0000 UTC m=+3615.174545244"
Oct 30 03:42:17.105154 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:17.105119    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:17.105643 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:17.105217    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:17.113430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:17.113404    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:42:20.977014 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:42:20.976971    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:42:39.114689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:39.114647    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:42:49.114694 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:49.114593    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:42:50.977718 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:42:50.977679    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:42:59.114311 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:42:59.114266    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:01.626730 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:01.626696    4334 scope.go:122] "RemoveContainer" containerID="2639a1f9ef4f9ff3421bb2e5353d0155afe9d5b1bf2708259a42a7af078ee086"
Oct 30 03:43:02.257644 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.257400    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:43:02.258220 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.258170    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="4a20a45d-98d9-4ee8-8c7f-c418f130ca22" containerName="otc-container" containerID="cri-o://32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37" gracePeriod=30
Oct 30 03:43:02.503232 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.503202    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:02.589910 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.589883    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-monitoring-tls-certs\") pod \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") "
Oct 30 03:43:02.590031 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.589936    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otlp-tls-certs\") pod \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") "
Oct 30 03:43:02.590031 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.590008    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otc-internal\") pod \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") "
Oct 30 03:43:02.590031 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.590025    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-kube-api-access-8l2fn\" (UniqueName: \"kubernetes.io/projected/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-kube-api-access-8l2fn\") pod \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\" (UID: \"4a20a45d-98d9-4ee8-8c7f-c418f130ca22\") "
Oct 30 03:43:02.590394 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.590363    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otc-internal" pod "4a20a45d-98d9-4ee8-8c7f-c418f130ca22" (UID: "4a20a45d-98d9-4ee8-8c7f-c418f130ca22"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:02.592121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.592090    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-kube-api-access-8l2fn" pod "4a20a45d-98d9-4ee8-8c7f-c418f130ca22" (UID: "4a20a45d-98d9-4ee8-8c7f-c418f130ca22"). InnerVolumeSpecName "kube-api-access-8l2fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:02.592121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.592102    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-monitoring-tls-certs" pod "4a20a45d-98d9-4ee8-8c7f-c418f130ca22" (UID: "4a20a45d-98d9-4ee8-8c7f-c418f130ca22"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:02.592391 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.592372    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otlp-tls-certs" pod "4a20a45d-98d9-4ee8-8c7f-c418f130ca22" (UID: "4a20a45d-98d9-4ee8-8c7f-c418f130ca22"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:02.691437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.691412    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.691437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.691436    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l2fn\" (UniqueName: \"kubernetes.io/projected/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-kube-api-access-8l2fn\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.691437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.691460    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.691889 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.691474    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a20a45d-98d9-4ee8-8c7f-c418f130ca22-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.746877 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.746846    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:02.747131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.747105    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo" containerID="cri-o://02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b" gracePeriod=30
Oct 30 03:43:02.747235 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.747133    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo-gateway" containerID="cri-o://9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca" gracePeriod=30
Oct 30 03:43:02.747296 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:02.747139    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo-gateway-opa" containerID="cri-o://89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604" gracePeriod=30
Oct 30 03:43:03.314838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.314809    4334 generic.go:356] "Generic (PLEG): container finished" podID="4a20a45d-98d9-4ee8-8c7f-c418f130ca22" containerID="32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37" exitCode=0
Oct 30 03:43:03.314951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.314846    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:03.314951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.314889    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4a20a45d-98d9-4ee8-8c7f-c418f130ca22","Type":"ContainerDied","Data":"32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37"}
Oct 30 03:43:03.314951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.314945    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4a20a45d-98d9-4ee8-8c7f-c418f130ca22","Type":"ContainerDied","Data":"cab878b328d32c7068d94f54c87517e0a6e03d7ed10983bd6f7ff115887d40f4"}
Oct 30 03:43:03.315097 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.314978    4334 scope.go:122] "RemoveContainer" containerID="32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37"
Oct 30 03:43:03.317263 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.317240    4334 generic.go:356] "Generic (PLEG): container finished" podID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerID="89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604" exitCode=0
Oct 30 03:43:03.317263 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.317264    4334 generic.go:356] "Generic (PLEG): container finished" podID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerID="02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b" exitCode=0
Oct 30 03:43:03.317427 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.317301    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e234bc2-d113-4b8e-b0a0-da497935ea20","Type":"ContainerDied","Data":"89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604"}
Oct 30 03:43:03.317427 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.317348    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e234bc2-d113-4b8e-b0a0-da497935ea20","Type":"ContainerDied","Data":"02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b"}
Oct 30 03:43:03.323252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.323234    4334 scope.go:122] "RemoveContainer" containerID="32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37"
Oct 30 03:43:03.323540 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:43:03.323522    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37\": container with ID starting with 32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37 not found: ID does not exist" containerID="32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37"
Oct 30 03:43:03.323604 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.323552    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37"} err="failed to get container status \"32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37\": rpc error: code = NotFound desc = could not find container \"32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37\": container with ID starting with 32cd113453cfda187b9948d73095b5f09edf3af7046665a7f771ec0b9e8f2a37 not found: ID does not exist"
Oct 30 03:43:03.367037 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.367016    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:43:03.380192 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.380169    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:43:03.984720 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:03.984694    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:04.102093 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102021    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102093 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102072    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-rbac\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102276 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102098    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-ca-bundle\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102276 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102206    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e234bc2-d113-4b8e-b0a0-da497935ea20-kube-api-access-9lnpl\" (UniqueName: \"kubernetes.io/projected/4e234bc2-d113-4b8e-b0a0-da497935ea20-kube-api-access-9lnpl\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102276    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-tenants\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102360    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102526 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102402    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-mtls\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102526 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102415    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-serving-cabundle" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:04.102526 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102439    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-conf\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102526 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102485    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-rbac" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:04.102526 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102503    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"4e234bc2-d113-4b8e-b0a0-da497935ea20\" (UID: \"4e234bc2-d113-4b8e-b0a0-da497935ea20\") "
Oct 30 03:43:04.102526 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102510    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-ca-bundle" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:04.102776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102715    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:04.102776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102734    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:04.102776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.102749    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:04.103079 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.103052    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-conf" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:04.104952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.104917    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-mtls" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:04.104952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.104946    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-tenants" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:04.105222 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.105206    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:43:04.105358 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.105340    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e234bc2-d113-4b8e-b0a0-da497935ea20-kube-api-access-9lnpl" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "kube-api-access-9lnpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:04.105417 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.105343    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "4e234bc2-d113-4b8e-b0a0-da497935ea20" (UID: "4e234bc2-d113-4b8e-b0a0-da497935ea20"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:04.203042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.203017    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lnpl\" (UniqueName: \"kubernetes.io/projected/4e234bc2-d113-4b8e-b0a0-da497935ea20-kube-api-access-9lnpl\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:04.203042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.203040    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-gateway-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:04.203182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.203066    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:43:04.203182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.203081    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:04.203182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.203096    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:04.203182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.203106    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e234bc2-d113-4b8e-b0a0-da497935ea20-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:43:04.210786 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.210773    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:43:04.304390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.304369    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\""
Oct 30 03:43:04.324590 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.324562    4334 generic.go:356] "Generic (PLEG): container finished" podID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerID="9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca" exitCode=0
Oct 30 03:43:04.324689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.324652    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:04.324689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.324652    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e234bc2-d113-4b8e-b0a0-da497935ea20","Type":"ContainerDied","Data":"9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca"}
Oct 30 03:43:04.324761 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.324710    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e234bc2-d113-4b8e-b0a0-da497935ea20","Type":"ContainerDied","Data":"c35e24b50eddcf08f56b7275e08203efa33bad56d9ea5ad42b63355d7f18965f"}
Oct 30 03:43:04.324761 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.324737    4334 scope.go:122] "RemoveContainer" containerID="89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604"
Oct 30 03:43:04.333099 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.333083    4334 scope.go:122] "RemoveContainer" containerID="9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca"
Oct 30 03:43:04.340511 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.340495    4334 scope.go:122] "RemoveContainer" containerID="02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b"
Oct 30 03:43:04.347488 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.347472    4334 scope.go:122] "RemoveContainer" containerID="89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604"
Oct 30 03:43:04.347758 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:43:04.347737    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604\": container with ID starting with 89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604 not found: ID does not exist" containerID="89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604"
Oct 30 03:43:04.347806 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.347767    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604"} err="failed to get container status \"89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604\": rpc error: code = NotFound desc = could not find container \"89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604\": container with ID starting with 89337d755d070a0503e778cb3c7937f1d36c1525d061d72a06257acc13921604 not found: ID does not exist"
Oct 30 03:43:04.347806 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.347783    4334 scope.go:122] "RemoveContainer" containerID="9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca"
Oct 30 03:43:04.348026 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:43:04.348007    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca\": container with ID starting with 9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca not found: ID does not exist" containerID="9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca"
Oct 30 03:43:04.348068 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.348035    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca"} err="failed to get container status \"9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca\": rpc error: code = NotFound desc = could not find container \"9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca\": container with ID starting with 9d541748b0ae21a347db16449649b83283d0541f1697c19ad1ea8ed8b17132ca not found: ID does not exist"
Oct 30 03:43:04.348068 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.348051    4334 scope.go:122] "RemoveContainer" containerID="02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b"
Oct 30 03:43:04.348309 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:43:04.348290    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b\": container with ID starting with 02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b not found: ID does not exist" containerID="02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b"
Oct 30 03:43:04.348349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.348315    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b"} err="failed to get container status \"02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b\": rpc error: code = NotFound desc = could not find container \"02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b\": container with ID starting with 02f2b05b0be4bc52b4cb03120aa766b72954dd63e3a5a74255edc7bc48df947b not found: ID does not exist"
Oct 30 03:43:04.380902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.380858    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:04.394339 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:04.394316    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:05.158585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:05.158547    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a20a45d-98d9-4ee8-8c7f-c418f130ca22" path="/var/lib/kubelet/pods/4a20a45d-98d9-4ee8-8c7f-c418f130ca22/volumes"
Oct 30 03:43:05.159142 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:05.159122    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" path="/var/lib/kubelet/pods/4e234bc2-d113-4b8e-b0a0-da497935ea20/volumes"
Oct 30 03:43:20.978897 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:43:20.978760    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:43:30.192047 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192010    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192291    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4a20a45d-98d9-4ee8-8c7f-c418f130ca22" containerName="otc-container"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192302    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a20a45d-98d9-4ee8-8c7f-c418f130ca22" containerName="otc-container"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192311    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo-gateway-opa"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192316    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo-gateway-opa"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192333    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo-gateway"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192338    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo-gateway"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192347    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192351    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192402    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a20a45d-98d9-4ee8-8c7f-c418f130ca22" containerName="otc-container"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192409    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo-gateway-opa"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192417    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo-gateway"
Oct 30 03:43:30.192678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.192423    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e234bc2-d113-4b8e-b0a0-da497935ea20" containerName="tempo"
Oct 30 03:43:30.195813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.195782    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.218864 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.218839    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:43:30.229127 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.229096    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:43:30.229272 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.229160    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:43:30.229272 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.229169    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\""
Oct 30 03:43:30.229272 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.229169    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7g6tv\""
Oct 30 03:43:30.229532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.229509    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:43:30.229604 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.229565    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:43:30.285635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.285600    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.285834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.285646    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g752f\" (UniqueName: \"kubernetes.io/projected/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-kube-api-access-g752f\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.285834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.285740    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.285834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.285806    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.386770 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.386730    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.386947 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.386786    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g752f\" (UniqueName: \"kubernetes.io/projected/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-kube-api-access-g752f\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.386947 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.386838    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.386947 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.386876    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.387600 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.387565    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.389476 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.389435    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.389476 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.389467    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.407785 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.407753    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g752f\" (UniqueName: \"kubernetes.io/projected/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-kube-api-access-g752f\") pod \"data-science-collector-collector-0\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.509777 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.509711    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:43:30.813324 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.813131    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:43:30.815873 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:43:30.815847    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43763ee4_4b6d_4c74_bb2b_6f150b0d9465.slice/crio-cbd45174a3bc6c47a6d6bfd2f6b6623e97fc9dc7959b55a476ebac45810cea71 WatchSource:0}: Error finding container cbd45174a3bc6c47a6d6bfd2f6b6623e97fc9dc7959b55a476ebac45810cea71: Status 404 returned error can't find the container with id cbd45174a3bc6c47a6d6bfd2f6b6623e97fc9dc7959b55a476ebac45810cea71
Oct 30 03:43:30.817810 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.817777    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:43:30.817916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:30.817834    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:43:31.445906 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:31.445870    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"43763ee4-4b6d-4c74-bb2b-6f150b0d9465","Type":"ContainerStarted","Data":"5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d"}
Oct 30 03:43:31.446327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:31.445920    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"43763ee4-4b6d-4c74-bb2b-6f150b0d9465","Type":"ContainerStarted","Data":"cbd45174a3bc6c47a6d6bfd2f6b6623e97fc9dc7959b55a476ebac45810cea71"}
Oct 30 03:43:31.507780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:31.507734    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.507721572 podStartE2EDuration="1.507721572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:31.506705856 +0000 UTC m=+3690.513296110" watchObservedRunningTime="2026-10-30 03:43:31.507721572 +0000 UTC m=+3690.514311825"
Oct 30 03:43:32.981694 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:32.981657    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"]
Oct 30 03:43:32.985109 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:32.985089    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.019341 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.019313    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\""
Oct 30 03:43:33.019517 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.019343    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\""
Oct 30 03:43:33.019517 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.019384    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-qj7ps\""
Oct 30 03:43:33.019517 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.019392    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\""
Oct 30 03:43:33.021469 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.021420    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"]
Oct 30 03:43:33.106669 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.106639    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-kube-api-access-cq5n9\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.106669 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.106677    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.106891 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.106712    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.106891 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.106825    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.106891 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.106857    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.175515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.175478    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"]
Oct 30 03:43:33.179256 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.179234    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.197510 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.197482    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"]
Oct 30 03:43:33.200741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.200721    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\""
Oct 30 03:43:33.200851 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.200836    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"s3-secret\""
Oct 30 03:43:33.208229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.208203    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-kube-api-access-cq5n9\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.208326 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.208248    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.208326 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.208292    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.208436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.208326    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.208517 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.208474    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.208783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.208761    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.209084 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.209063    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.209343 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.209323    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.211041 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.211018    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.214932 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.214903    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"]
Oct 30 03:43:33.218322 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.218293    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.229081 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.229057    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-kube-api-access-cq5n9\") pod \"tempo-data-science-tempostack-distributor-6d55f47b85-s8fff\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.237531 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.237483    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"]
Oct 30 03:43:33.279677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.279652    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"]
Oct 30 03:43:33.282808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.282791    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.293665 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.293642    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\""
Oct 30 03:43:33.298813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.298791    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:33.301498 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.301474    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"]
Oct 30 03:43:33.303903 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.303883    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\""
Oct 30 03:43:33.309214 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309191    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4b9\" (UniqueName: \"kubernetes.io/projected/1ed9f214-cff6-45da-89c4-165615d1a33e-kube-api-access-jj4b9\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.309304 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309227    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.309304 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309252    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-conf\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.309417 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309323    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.309503 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309426    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.309591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309565    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.309666 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309644    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.309728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309689    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.309778 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309723    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56pf2\" (UniqueName: \"kubernetes.io/projected/84c16162-e73b-47b2-8666-54a29cb405ee-kube-api-access-56pf2\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.309841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.309795    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.411224 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411186    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.411375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411230    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.411375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411257    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56pf2\" (UniqueName: \"kubernetes.io/projected/84c16162-e73b-47b2-8666-54a29cb405ee-kube-api-access-56pf2\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.411375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411284    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.411375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411323    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.411637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411553    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j2m6\" (UniqueName: \"kubernetes.io/projected/178d490a-506f-4f04-b987-abb5e80b93e5-kube-api-access-9j2m6\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.411637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411615    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.411751 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411675    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.411751 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411679    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.411751 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411713    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.411925 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411788    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4b9\" (UniqueName: \"kubernetes.io/projected/1ed9f214-cff6-45da-89c4-165615d1a33e-kube-api-access-jj4b9\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.411925 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411826    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.411925 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411864    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-conf\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.411925 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411905    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.412131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411963    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.412131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.411994    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.412131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.412082    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.412131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.412115    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.412131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.412124    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.412760 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.412738    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-conf\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.413017 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.412993    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.414502 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.414476    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.414589 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.414508    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.431825 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.431806    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56pf2\" (UniqueName: \"kubernetes.io/projected/84c16162-e73b-47b2-8666-54a29cb405ee-kube-api-access-56pf2\") pod \"tempo-data-science-tempostack-querier-579c44f4c4-zldk6\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.432030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.432011    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4b9\" (UniqueName: \"kubernetes.io/projected/1ed9f214-cff6-45da-89c4-165615d1a33e-kube-api-access-jj4b9\") pod \"tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.492624 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.492574    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:33.512653 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.512632    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j2m6\" (UniqueName: \"kubernetes.io/projected/178d490a-506f-4f04-b987-abb5e80b93e5-kube-api-access-9j2m6\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.512735 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.512668    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.512735 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.512694    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.512840 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.512813    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.512964 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.512947    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.513294 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.513264    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.513429 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.513409    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.513541 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.513524    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.515020 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.514999    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.531317 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.530992    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:33.533586 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.533567    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j2m6\" (UniqueName: \"kubernetes.io/projected/178d490a-506f-4f04-b987-abb5e80b93e5-kube-api-access-9j2m6\") pod \"tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.594754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.594721    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:33.620089 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.620045    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"]
Oct 30 03:43:33.788800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.788733    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"]
Oct 30 03:43:33.792348 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:43:33.792315    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed9f214_cff6_45da_89c4_165615d1a33e.slice/crio-aa5bbf3e5c9d40eff50d022f8d2dbbb18317f503bee6d18e9d83fdd2cf23a7a9 WatchSource:0}: Error finding container aa5bbf3e5c9d40eff50d022f8d2dbbb18317f503bee6d18e9d83fdd2cf23a7a9: Status 404 returned error can't find the container with id aa5bbf3e5c9d40eff50d022f8d2dbbb18317f503bee6d18e9d83fdd2cf23a7a9
Oct 30 03:43:33.844978 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.844953    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"]
Oct 30 03:43:33.846268 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:43:33.846242    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c16162_e73b_47b2_8666_54a29cb405ee.slice/crio-3032de51e35132fdafc2e62dc3d20ff13a18ec775f127e08282150ca9630fb6e WatchSource:0}: Error finding container 3032de51e35132fdafc2e62dc3d20ff13a18ec775f127e08282150ca9630fb6e: Status 404 returned error can't find the container with id 3032de51e35132fdafc2e62dc3d20ff13a18ec775f127e08282150ca9630fb6e
Oct 30 03:43:33.905794 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:33.905771    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"]
Oct 30 03:43:33.966940 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:43:33.966910    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178d490a_506f_4f04_b987_abb5e80b93e5.slice/crio-1fed18b1f5da80e179c05f2c7fe3b41a64eac9f3207e8c6a06fdc34e47eb0afb WatchSource:0}: Error finding container 1fed18b1f5da80e179c05f2c7fe3b41a64eac9f3207e8c6a06fdc34e47eb0afb: Status 404 returned error can't find the container with id 1fed18b1f5da80e179c05f2c7fe3b41a64eac9f3207e8c6a06fdc34e47eb0afb
Oct 30 03:43:34.059079 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.059011    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"]
Oct 30 03:43:34.062677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.062658    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.080934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.080797    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\""
Oct 30 03:43:34.083364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.083337    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"]
Oct 30 03:43:34.084746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.084724    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-brrkd\""
Oct 30 03:43:34.084876 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.084815    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:43:34.084972 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.084902    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\""
Oct 30 03:43:34.181111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.181077    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\""
Oct 30 03:43:34.181111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.181076    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:43:34.219675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.219647    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.219812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.219726    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-cabundle\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.219812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.219752    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.219812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.219779    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-serving-certs\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.219937 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.219811    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2xx\" (UniqueName: \"kubernetes.io/projected/91b838ba-d00d-4095-8d68-823976049cf5-kube-api-access-pk2xx\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.219937 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.219912    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-rbac\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.220016 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.219953    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tenant\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.320531 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.320474    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-rbac\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.320531 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.320519    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tenant\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.320675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.320572    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.320675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.320600    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-cabundle\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.320675 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.320644    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.320837 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.320677    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-serving-certs\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.320837 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.320724    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2xx\" (UniqueName: \"kubernetes.io/projected/91b838ba-d00d-4095-8d68-823976049cf5-kube-api-access-pk2xx\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.321222 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.321200    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-rbac\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.321308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.321275    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.321390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.321369    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-cabundle\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.322868 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.322839    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.323222 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.323206    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tenant\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.323377 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.323360    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-serving-certs\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.341289 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.341267    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2xx\" (UniqueName: \"kubernetes.io/projected/91b838ba-d00d-4095-8d68-823976049cf5-kube-api-access-pk2xx\") pod \"tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.393179 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.393154    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.478108    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" event={"ID":"84c16162-e73b-47b2-8666-54a29cb405ee","Type":"ContainerStarted","Data":"8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f"}
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.478161    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" event={"ID":"84c16162-e73b-47b2-8666-54a29cb405ee","Type":"ContainerStarted","Data":"3032de51e35132fdafc2e62dc3d20ff13a18ec775f127e08282150ca9630fb6e"}
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.478810    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.480387    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" event={"ID":"178d490a-506f-4f04-b987-abb5e80b93e5","Type":"ContainerStarted","Data":"456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91"}
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.480431    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" event={"ID":"178d490a-506f-4f04-b987-abb5e80b93e5","Type":"ContainerStarted","Data":"1fed18b1f5da80e179c05f2c7fe3b41a64eac9f3207e8c6a06fdc34e47eb0afb"}
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.480491    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.482009    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" event={"ID":"1ed9f214-cff6-45da-89c4-165615d1a33e","Type":"ContainerStarted","Data":"329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456"}
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.482045    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" event={"ID":"1ed9f214-cff6-45da-89c4-165615d1a33e","Type":"ContainerStarted","Data":"aa5bbf3e5c9d40eff50d022f8d2dbbb18317f503bee6d18e9d83fdd2cf23a7a9"}
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.482436    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.483709    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff" event={"ID":"cafb0bb0-5835-4a9c-982c-14bcb4765b5a","Type":"ContainerStarted","Data":"0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb"}
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.483742    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff" event={"ID":"cafb0bb0-5835-4a9c-982c-14bcb4765b5a","Type":"ContainerStarted","Data":"48d3c786c5bcf43b69beff1dae4817c8d18296028eb2917e096203a70c3ce52b"}
Oct 30 03:43:34.484483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.483762    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:34.522803 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.522760    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" podStartSLOduration=1.522747473 podStartE2EDuration="1.522747473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:34.521018409 +0000 UTC m=+3693.527608663" watchObservedRunningTime="2026-10-30 03:43:34.522747473 +0000 UTC m=+3693.529337727"
Oct 30 03:43:34.563852 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.563809    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" podStartSLOduration=1.5637975050000001 podStartE2EDuration="1.563797505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:34.562404035 +0000 UTC m=+3693.568994304" watchObservedRunningTime="2026-10-30 03:43:34.563797505 +0000 UTC m=+3693.570387759"
Oct 30 03:43:34.604781 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.604693    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff" podStartSLOduration=2.60468001 podStartE2EDuration="2.60468001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:34.603381012 +0000 UTC m=+3693.609971267" watchObservedRunningTime="2026-10-30 03:43:34.60468001 +0000 UTC m=+3693.611270264"
Oct 30 03:43:34.646713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.646671    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" podStartSLOduration=1.646660277 podStartE2EDuration="1.646660277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:34.644623479 +0000 UTC m=+3693.651213730" watchObservedRunningTime="2026-10-30 03:43:34.646660277 +0000 UTC m=+3693.653250530"
Oct 30 03:43:34.725335 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:34.725313    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"]
Oct 30 03:43:34.727170 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:43:34.727150    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b838ba_d00d_4095_8d68_823976049cf5.slice/crio-5903896711ee67fff22c5a190195e0df240ee34250c700c8d249e85523259d58 WatchSource:0}: Error finding container 5903896711ee67fff22c5a190195e0df240ee34250c700c8d249e85523259d58: Status 404 returned error can't find the container with id 5903896711ee67fff22c5a190195e0df240ee34250c700c8d249e85523259d58
Oct 30 03:43:35.490898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:35.490808    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" event={"ID":"91b838ba-d00d-4095-8d68-823976049cf5","Type":"ContainerStarted","Data":"ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7"}
Oct 30 03:43:35.490898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:35.490880    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" event={"ID":"91b838ba-d00d-4095-8d68-823976049cf5","Type":"ContainerStarted","Data":"7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2"}
Oct 30 03:43:35.491418 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:35.490911    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" event={"ID":"91b838ba-d00d-4095-8d68-823976049cf5","Type":"ContainerStarted","Data":"5903896711ee67fff22c5a190195e0df240ee34250c700c8d249e85523259d58"}
Oct 30 03:43:35.533850 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:35.533797    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" podStartSLOduration=1.533784873 podStartE2EDuration="1.533784873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:35.533214191 +0000 UTC m=+3694.539804446" watchObservedRunningTime="2026-10-30 03:43:35.533784873 +0000 UTC m=+3694.540375127"
Oct 30 03:43:36.068967 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.068926    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:43:36.072557 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.072538    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.088509 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.088485    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\""
Oct 30 03:43:36.103673 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.103648    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:43:36.236946 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.236915    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.236946 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.236949    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.237116 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.236979    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8cmp\" (UniqueName: \"kubernetes.io/projected/c8e33605-87f4-48ea-b630-060470e9a8f0-kube-api-access-m8cmp\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.237173 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.237157    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.338374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.338305    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.338514 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.338371    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.338514 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.338402    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.338514 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.338465    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8cmp\" (UniqueName: \"kubernetes.io/projected/c8e33605-87f4-48ea-b630-060470e9a8f0-kube-api-access-m8cmp\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.338998 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.338973    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.339155 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.339138    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.340790 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.340769    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.359580 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.359558    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8cmp\" (UniqueName: \"kubernetes.io/projected/c8e33605-87f4-48ea-b630-060470e9a8f0-kube-api-access-m8cmp\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:36.495901 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.495876    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:36.496337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.495931    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:36.497106 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.497077    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.73:8083/ready\": dial tcp 10.133.0.73:8083: connect: connection refused"
Oct 30 03:43:36.967797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:36.967763    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:37.565044 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:37.565009    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:43:38.560174 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:38.560143    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:38.591873 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:38.591843    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:38.660941 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:38.660908    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:38.661121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:38.661048    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:38.676344 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:38.676322    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") DevicePath \"csi-41cb6fd7406043f5cdbef35d658cd5b6e8fa4caf7266326e58ff992d85d9d51c\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:38.897628 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:38.897599    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/239011135fcbd13577b023b82456a4d813e5268c79848b272c29ac9f9cbb7c39/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:38.924699 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:38.924670    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:39.084857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:39.084822    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:39.397798 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:39.397774    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:43:39.400173 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:43:39.400131    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e33605_87f4_48ea_b630_060470e9a8f0.slice/crio-bcc0994c7dcad9a99e31c1575322b8f15953a9f16c02d18cdcca348f62c8e95a WatchSource:0}: Error finding container bcc0994c7dcad9a99e31c1575322b8f15953a9f16c02d18cdcca348f62c8e95a: Status 404 returned error can't find the container with id bcc0994c7dcad9a99e31c1575322b8f15953a9f16c02d18cdcca348f62c8e95a
Oct 30 03:43:39.511176 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:39.511142    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"c8e33605-87f4-48ea-b630-060470e9a8f0","Type":"ContainerStarted","Data":"f04abe4f5d998b38a81a2c4b66a98e52ea8e5bdf357e9c629dfb89ebfad4e9a2"}
Oct 30 03:43:39.511334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:39.511200    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"c8e33605-87f4-48ea-b630-060470e9a8f0","Type":"ContainerStarted","Data":"bcc0994c7dcad9a99e31c1575322b8f15953a9f16c02d18cdcca348f62c8e95a"}
Oct 30 03:43:39.511489 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:39.511442    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:43:39.555560 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:39.555511    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=6.555500285 podStartE2EDuration="6.555500285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:39.554421938 +0000 UTC m=+3698.561012193" watchObservedRunningTime="2026-10-30 03:43:39.555500285 +0000 UTC m=+3698.562090538"
Oct 30 03:43:50.978121 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:43:50.978081    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:43:55.497098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:55.497069    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:43:55.497562 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:55.497481    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:43:56.501625 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:56.501596    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:43:56.665111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:43:56.665078    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:44:00.521819 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:00.521776    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:05.496023 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:05.495981    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:10.520820 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:10.520783    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:15.495338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:15.495241    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:20.520462 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:20.520419    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:44:20.978206 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:44:20.978176    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:44:25.495575 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:25.495540    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:35.495987 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:35.495960    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:44:41.342216 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.342187    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:41.342607 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.342380    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="43763ee4-4b6d-4c74-bb2b-6f150b0d9465" containerName="otc-container" containerID="cri-o://5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d" gracePeriod=30
Oct 30 03:44:41.589512 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.589486    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:41.777037 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.777006    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otlp-tls-certs\") pod \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") "
Oct 30 03:44:41.777230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.777057    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-kube-api-access-g752f\" (UniqueName: \"kubernetes.io/projected/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-kube-api-access-g752f\") pod \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") "
Oct 30 03:44:41.777230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.777104    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-monitoring-tls-certs\") pod \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") "
Oct 30 03:44:41.777230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.777129    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otc-internal\" (UniqueName: \"kubernetes.io/configmap/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otc-internal\") pod \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\" (UID: \"43763ee4-4b6d-4c74-bb2b-6f150b0d9465\") "
Oct 30 03:44:41.777604 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.777574    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otc-internal" pod "43763ee4-4b6d-4c74-bb2b-6f150b0d9465" (UID: "43763ee4-4b6d-4c74-bb2b-6f150b0d9465"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:41.779138 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.779111    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otlp-tls-certs" pod "43763ee4-4b6d-4c74-bb2b-6f150b0d9465" (UID: "43763ee4-4b6d-4c74-bb2b-6f150b0d9465"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:41.779244 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.779138    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-monitoring-tls-certs" pod "43763ee4-4b6d-4c74-bb2b-6f150b0d9465" (UID: "43763ee4-4b6d-4c74-bb2b-6f150b0d9465"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:41.779244 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.779162    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-kube-api-access-g752f" pod "43763ee4-4b6d-4c74-bb2b-6f150b0d9465" (UID: "43763ee4-4b6d-4c74-bb2b-6f150b0d9465"). InnerVolumeSpecName "kube-api-access-g752f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:41.830595 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.830562    4334 generic.go:356] "Generic (PLEG): container finished" podID="43763ee4-4b6d-4c74-bb2b-6f150b0d9465" containerID="5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d" exitCode=0
Oct 30 03:44:41.830695 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.830610    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:41.830695 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.830639    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"43763ee4-4b6d-4c74-bb2b-6f150b0d9465","Type":"ContainerDied","Data":"5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d"}
Oct 30 03:44:41.830695 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.830691    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"43763ee4-4b6d-4c74-bb2b-6f150b0d9465","Type":"ContainerDied","Data":"cbd45174a3bc6c47a6d6bfd2f6b6623e97fc9dc7959b55a476ebac45810cea71"}
Oct 30 03:44:41.830813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.830713    4334 scope.go:122] "RemoveContainer" containerID="5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d"
Oct 30 03:44:41.839270 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.839248    4334 scope.go:122] "RemoveContainer" containerID="5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d"
Oct 30 03:44:41.839554 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:44:41.839535    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d\": container with ID starting with 5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d not found: ID does not exist" containerID="5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d"
Oct 30 03:44:41.839625 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.839562    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d"} err="failed to get container status \"5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d\": rpc error: code = NotFound desc = could not find container \"5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d\": container with ID starting with 5e3f953d4f7196a94957a0876c70eb3023402b767fd657e84bbc7b11cb77f29d not found: ID does not exist"
Oct 30 03:44:41.877862 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.877844    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:41.877952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.877864    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g752f\" (UniqueName: \"kubernetes.io/projected/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-kube-api-access-g752f\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:41.877952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.877874    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:41.877952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.877883    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43763ee4-4b6d-4c74-bb2b-6f150b0d9465-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:41.892420 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.892395    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:41.901815 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.901793    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:41.984434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.984403    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:41.984828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.984811    4334 cpu_manager.go:414] "containerMap: removing container" podUID="43763ee4-4b6d-4c74-bb2b-6f150b0d9465" containerName="otc-container"
Oct 30 03:44:41.984906 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.984828    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43763ee4-4b6d-4c74-bb2b-6f150b0d9465" containerName="otc-container"
Oct 30 03:44:41.984962 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.984932    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="43763ee4-4b6d-4c74-bb2b-6f150b0d9465" containerName="otc-container"
Oct 30 03:44:41.989159 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:41.989140    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.018532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.018506    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:42.019848 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.019829    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\""
Oct 30 03:44:42.020199 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.020180    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7g6tv\""
Oct 30 03:44:42.020294 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.020257    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:44:42.027333 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.027285    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:42.180325 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.180299    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzj5\" (UniqueName: \"kubernetes.io/projected/1e92fff2-f048-484c-92a9-daf34f1e84ac-kube-api-access-4jzj5\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.180557 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.180336    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.180557 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.180360    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.180557 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.180472    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e92fff2-f048-484c-92a9-daf34f1e84ac-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.281467 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.281379    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzj5\" (UniqueName: \"kubernetes.io/projected/1e92fff2-f048-484c-92a9-daf34f1e84ac-kube-api-access-4jzj5\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.281467 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.281425    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.281670 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.281484    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.281670 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.281509    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e92fff2-f048-484c-92a9-daf34f1e84ac-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.282238 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.282215    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e92fff2-f048-484c-92a9-daf34f1e84ac-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.283937 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.283913    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.284046 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.283955    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.302207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.302184    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzj5\" (UniqueName: \"kubernetes.io/projected/1e92fff2-f048-484c-92a9-daf34f1e84ac-kube-api-access-4jzj5\") pod \"data-science-collector-collector-0\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.560066 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.559980    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:42.560507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.560223    4334 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-7g6tv\" not found"
Oct 30 03:44:42.560507 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.560275    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:42.870384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.870316    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:42.872851 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:44:42.872825    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e92fff2_f048_484c_92a9_daf34f1e84ac.slice/crio-053f5b038ab7a774f415d6d814c40aa1f96c7bce9dedcf03d03c68f0a873f4a6 WatchSource:0}: Error finding container 053f5b038ab7a774f415d6d814c40aa1f96c7bce9dedcf03d03c68f0a873f4a6: Status 404 returned error can't find the container with id 053f5b038ab7a774f415d6d814c40aa1f96c7bce9dedcf03d03c68f0a873f4a6
Oct 30 03:44:42.875254 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.875219    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:44:42.875340 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:42.875281    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:44:43.171515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.171483    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43763ee4-4b6d-4c74-bb2b-6f150b0d9465" path="/var/lib/kubelet/pods/43763ee4-4b6d-4c74-bb2b-6f150b0d9465/volumes"
Oct 30 03:44:43.270203 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.270160    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:44:43.270430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.270404    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerName="tempo" containerID="cri-o://f04abe4f5d998b38a81a2c4b66a98e52ea8e5bdf357e9c629dfb89ebfad4e9a2" gracePeriod=30
Oct 30 03:44:43.281980 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.281954    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"]
Oct 30 03:44:43.282200 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.282175    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway" containerID="cri-o://7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2" gracePeriod=30
Oct 30 03:44:43.282297 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.282222    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway-opa" containerID="cri-o://ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7" gracePeriod=30
Oct 30 03:44:43.285598 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.285575    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"]
Oct 30 03:44:43.285782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.285763    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff" podUID="cafb0bb0-5835-4a9c-982c-14bcb4765b5a" containerName="tempo" containerID="cri-o://0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb" gracePeriod=30
Oct 30 03:44:43.287030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.287005    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"]
Oct 30 03:44:43.287180 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.287160    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" podUID="1ed9f214-cff6-45da-89c4-165615d1a33e" containerName="tempo" containerID="cri-o://329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456" gracePeriod=30
Oct 30 03:44:43.291585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.291510    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"]
Oct 30 03:44:43.291710 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.291688    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" podUID="84c16162-e73b-47b2-8666-54a29cb405ee" containerName="tempo" containerID="cri-o://8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f" gracePeriod=30
Oct 30 03:44:43.297422 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.297398    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"]
Oct 30 03:44:43.297585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.297567    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo" containerID="cri-o://456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91" gracePeriod=30
Oct 30 03:44:43.842162 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.842123    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1e92fff2-f048-484c-92a9-daf34f1e84ac","Type":"ContainerStarted","Data":"c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a"}
Oct 30 03:44:43.842638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.842143    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="1e92fff2-f048-484c-92a9-daf34f1e84ac" containerName="otc-container" containerID="cri-o://c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a" gracePeriod=30
Oct 30 03:44:43.842638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.842191    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1e92fff2-f048-484c-92a9-daf34f1e84ac","Type":"ContainerStarted","Data":"053f5b038ab7a774f415d6d814c40aa1f96c7bce9dedcf03d03c68f0a873f4a6"}
Oct 30 03:44:43.844215 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.844192    4334 generic.go:356] "Generic (PLEG): container finished" podID="91b838ba-d00d-4095-8d68-823976049cf5" containerID="ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7" exitCode=0
Oct 30 03:44:43.844308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:43.844238    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" event={"ID":"91b838ba-d00d-4095-8d68-823976049cf5","Type":"ContainerDied","Data":"ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7"}
Oct 30 03:44:44.126906 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.126882    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:44:44.129763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.129742    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:44.183084 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.183043    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=3.183030677 podStartE2EDuration="3.183030677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:43.895127908 +0000 UTC m=+3762.901718162" watchObservedRunningTime="2026-10-30 03:44:44.183030677 +0000 UTC m=+3763.189620931"
Oct 30 03:44:44.198497 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198479    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e92fff2-f048-484c-92a9-daf34f1e84ac-kube-api-access-4jzj5\" (UniqueName: \"kubernetes.io/projected/1e92fff2-f048-484c-92a9-daf34f1e84ac-kube-api-access-4jzj5\") pod \"1e92fff2-f048-484c-92a9-daf34f1e84ac\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") "
Oct 30 03:44:44.198622 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198520    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-kube-api-access-cq5n9\") pod \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") "
Oct 30 03:44:44.198622 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198551    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-distributor-mtls\") pod \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") "
Oct 30 03:44:44.198622 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198590    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-monitoring-tls-certs\") pod \"1e92fff2-f048-484c-92a9-daf34f1e84ac\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") "
Oct 30 03:44:44.198622 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198614    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-ca-bundle\") pod \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") "
Oct 30 03:44:44.198838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198650    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-otlp-tls-certs\") pod \"1e92fff2-f048-484c-92a9-daf34f1e84ac\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") "
Oct 30 03:44:44.198838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198683    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-tmp-storage\") pod \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") "
Oct 30 03:44:44.198838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198711    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e92fff2-f048-484c-92a9-daf34f1e84ac-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e92fff2-f048-484c-92a9-daf34f1e84ac-otc-internal\") pod \"1e92fff2-f048-484c-92a9-daf34f1e84ac\" (UID: \"1e92fff2-f048-484c-92a9-daf34f1e84ac\") "
Oct 30 03:44:44.198838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.198781    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-conf\") pod \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\" (UID: \"cafb0bb0-5835-4a9c-982c-14bcb4765b5a\") "
Oct 30 03:44:44.199712 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.199681    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e92fff2-f048-484c-92a9-daf34f1e84ac-otc-internal" pod "1e92fff2-f048-484c-92a9-daf34f1e84ac" (UID: "1e92fff2-f048-484c-92a9-daf34f1e84ac"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:44.199817 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.199759    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-tmp-storage" pod "cafb0bb0-5835-4a9c-982c-14bcb4765b5a" (UID: "cafb0bb0-5835-4a9c-982c-14bcb4765b5a"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:44:44.199817 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.199802    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-ca-bundle" pod "cafb0bb0-5835-4a9c-982c-14bcb4765b5a" (UID: "cafb0bb0-5835-4a9c-982c-14bcb4765b5a"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:44.199935 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.199853    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-conf" pod "cafb0bb0-5835-4a9c-982c-14bcb4765b5a" (UID: "cafb0bb0-5835-4a9c-982c-14bcb4765b5a"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:44.200962 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.200941    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-kube-api-access-cq5n9" pod "cafb0bb0-5835-4a9c-982c-14bcb4765b5a" (UID: "cafb0bb0-5835-4a9c-982c-14bcb4765b5a"). InnerVolumeSpecName "kube-api-access-cq5n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:44.201155 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.201131    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e92fff2-f048-484c-92a9-daf34f1e84ac-kube-api-access-4jzj5" pod "1e92fff2-f048-484c-92a9-daf34f1e84ac" (UID: "1e92fff2-f048-484c-92a9-daf34f1e84ac"). InnerVolumeSpecName "kube-api-access-4jzj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:44.201336 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.201311    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-distributor-mtls" pod "cafb0bb0-5835-4a9c-982c-14bcb4765b5a" (UID: "cafb0bb0-5835-4a9c-982c-14bcb4765b5a"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.201426 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.201360    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-otlp-tls-certs" pod "1e92fff2-f048-484c-92a9-daf34f1e84ac" (UID: "1e92fff2-f048-484c-92a9-daf34f1e84ac"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.201841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.201826    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-monitoring-tls-certs" pod "1e92fff2-f048-484c-92a9-daf34f1e84ac" (UID: "1e92fff2-f048-484c-92a9-daf34f1e84ac"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.300075 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300040    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5n9\" (UniqueName: \"kubernetes.io/projected/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-kube-api-access-cq5n9\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.300075 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300073    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.300280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300086    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.300280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300100    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.300280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300112    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e92fff2-f048-484c-92a9-daf34f1e84ac-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.300280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300123    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-tmp-storage\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.300280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300134    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e92fff2-f048-484c-92a9-daf34f1e84ac-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.300280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300145    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cafb0bb0-5835-4a9c-982c-14bcb4765b5a-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.300280 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.300156    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jzj5\" (UniqueName: \"kubernetes.io/projected/1e92fff2-f048-484c-92a9-daf34f1e84ac-kube-api-access-4jzj5\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.630331 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.630304    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:44:44.702775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.702747    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-ca-bundle\") pod \"91b838ba-d00d-4095-8d68-823976049cf5\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") "
Oct 30 03:44:44.702775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.702779    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tenant\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tenant\") pod \"91b838ba-d00d-4095-8d68-823976049cf5\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") "
Oct 30 03:44:44.702957 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.702802    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-gateway-mtls\") pod \"91b838ba-d00d-4095-8d68-823976049cf5\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") "
Oct 30 03:44:44.702957 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.702830    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b838ba-d00d-4095-8d68-823976049cf5-kube-api-access-pk2xx\" (UniqueName: \"kubernetes.io/projected/91b838ba-d00d-4095-8d68-823976049cf5-kube-api-access-pk2xx\") pod \"91b838ba-d00d-4095-8d68-823976049cf5\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") "
Oct 30 03:44:44.703062 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.702948    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-cabundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-cabundle\") pod \"91b838ba-d00d-4095-8d68-823976049cf5\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") "
Oct 30 03:44:44.703062 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.702990    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-serving-certs\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-serving-certs\") pod \"91b838ba-d00d-4095-8d68-823976049cf5\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") "
Oct 30 03:44:44.703062 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.703056    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-rbac\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-rbac\") pod \"91b838ba-d00d-4095-8d68-823976049cf5\" (UID: \"91b838ba-d00d-4095-8d68-823976049cf5\") "
Oct 30 03:44:44.703322 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.703266    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-ca-bundle" pod "91b838ba-d00d-4095-8d68-823976049cf5" (UID: "91b838ba-d00d-4095-8d68-823976049cf5"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:44.703600 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.703340    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-cabundle" pod "91b838ba-d00d-4095-8d68-823976049cf5" (UID: "91b838ba-d00d-4095-8d68-823976049cf5"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:44.703600 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.703560    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-rbac" pod "91b838ba-d00d-4095-8d68-823976049cf5" (UID: "91b838ba-d00d-4095-8d68-823976049cf5"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:44.704974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.704946    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b838ba-d00d-4095-8d68-823976049cf5-kube-api-access-pk2xx" pod "91b838ba-d00d-4095-8d68-823976049cf5" (UID: "91b838ba-d00d-4095-8d68-823976049cf5"). InnerVolumeSpecName "kube-api-access-pk2xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:44.705136 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.705117    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-gateway-mtls" pod "91b838ba-d00d-4095-8d68-823976049cf5" (UID: "91b838ba-d00d-4095-8d68-823976049cf5"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.705374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.705355    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-serving-certs" pod "91b838ba-d00d-4095-8d68-823976049cf5" (UID: "91b838ba-d00d-4095-8d68-823976049cf5"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.708780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.708761    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tenant" pod "91b838ba-d00d-4095-8d68-823976049cf5" (UID: "91b838ba-d00d-4095-8d68-823976049cf5"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.803712 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.803685    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.803712 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.803707    4334 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tenant\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.803832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.803717    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.803832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.803726    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2xx\" (UniqueName: \"kubernetes.io/projected/91b838ba-d00d-4095-8d68-823976049cf5-kube-api-access-pk2xx\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.803832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.803735    4334 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.803832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.803765    4334 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/91b838ba-d00d-4095-8d68-823976049cf5-serving-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.803832 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.803773    4334 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/91b838ba-d00d-4095-8d68-823976049cf5-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.850337 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.850312    4334 generic.go:356] "Generic (PLEG): container finished" podID="1e92fff2-f048-484c-92a9-daf34f1e84ac" containerID="c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a" exitCode=0
Oct 30 03:44:44.850727 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.850354    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:44:44.850727 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.850394    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1e92fff2-f048-484c-92a9-daf34f1e84ac","Type":"ContainerDied","Data":"c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a"}
Oct 30 03:44:44.850727 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.850443    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1e92fff2-f048-484c-92a9-daf34f1e84ac","Type":"ContainerDied","Data":"053f5b038ab7a774f415d6d814c40aa1f96c7bce9dedcf03d03c68f0a873f4a6"}
Oct 30 03:44:44.850727 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.850492    4334 scope.go:122] "RemoveContainer" containerID="c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a"
Oct 30 03:44:44.851911 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.851886    4334 generic.go:356] "Generic (PLEG): container finished" podID="cafb0bb0-5835-4a9c-982c-14bcb4765b5a" containerID="0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb" exitCode=0
Oct 30 03:44:44.851988 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.851937    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"
Oct 30 03:44:44.851988 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.851972    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff" event={"ID":"cafb0bb0-5835-4a9c-982c-14bcb4765b5a","Type":"ContainerDied","Data":"0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb"}
Oct 30 03:44:44.852122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.852015    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff" event={"ID":"cafb0bb0-5835-4a9c-982c-14bcb4765b5a","Type":"ContainerDied","Data":"48d3c786c5bcf43b69beff1dae4817c8d18296028eb2917e096203a70c3ce52b"}
Oct 30 03:44:44.853840 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.853815    4334 generic.go:356] "Generic (PLEG): container finished" podID="91b838ba-d00d-4095-8d68-823976049cf5" containerID="7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2" exitCode=0
Oct 30 03:44:44.853957 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.853852    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" event={"ID":"91b838ba-d00d-4095-8d68-823976049cf5","Type":"ContainerDied","Data":"7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2"}
Oct 30 03:44:44.853957 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.853880    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"
Oct 30 03:44:44.853957 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.853890    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4" event={"ID":"91b838ba-d00d-4095-8d68-823976049cf5","Type":"ContainerDied","Data":"5903896711ee67fff22c5a190195e0df240ee34250c700c8d249e85523259d58"}
Oct 30 03:44:44.859366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.859288    4334 scope.go:122] "RemoveContainer" containerID="c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a"
Oct 30 03:44:44.859654 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:44:44.859633    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a\": container with ID starting with c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a not found: ID does not exist" containerID="c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a"
Oct 30 03:44:44.859753 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.859663    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a"} err="failed to get container status \"c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a\": rpc error: code = NotFound desc = could not find container \"c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a\": container with ID starting with c08b351465d37cb41bf517d626710cb51709b9e497d3188b418facc893011a9a not found: ID does not exist"
Oct 30 03:44:44.859753 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.859687    4334 scope.go:122] "RemoveContainer" containerID="0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb"
Oct 30 03:44:44.868976 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.868960    4334 scope.go:122] "RemoveContainer" containerID="0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb"
Oct 30 03:44:44.869247 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:44:44.869227    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb\": container with ID starting with 0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb not found: ID does not exist" containerID="0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb"
Oct 30 03:44:44.869298 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.869255    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb"} err="failed to get container status \"0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb\": rpc error: code = NotFound desc = could not find container \"0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb\": container with ID starting with 0766b54f9fd14608a75939b0667ea40c42177d91ae7744557c51d310b75f9fbb not found: ID does not exist"
Oct 30 03:44:44.869298 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.869270    4334 scope.go:122] "RemoveContainer" containerID="ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7"
Oct 30 03:44:44.876287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.876273    4334 scope.go:122] "RemoveContainer" containerID="7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2"
Oct 30 03:44:44.883649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.883635    4334 scope.go:122] "RemoveContainer" containerID="ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7"
Oct 30 03:44:44.883881 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:44:44.883866    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7\": container with ID starting with ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7 not found: ID does not exist" containerID="ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7"
Oct 30 03:44:44.883928 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.883887    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7"} err="failed to get container status \"ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7\": rpc error: code = NotFound desc = could not find container \"ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7\": container with ID starting with ffd5d92d79bed783dd23b4b0ca509a221be2eb522f4ca668d790a06abf223bc7 not found: ID does not exist"
Oct 30 03:44:44.883928 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.883902    4334 scope.go:122] "RemoveContainer" containerID="7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2"
Oct 30 03:44:44.884103 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:44:44.884084    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2\": container with ID starting with 7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2 not found: ID does not exist" containerID="7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2"
Oct 30 03:44:44.884146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.884111    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2"} err="failed to get container status \"7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2\": rpc error: code = NotFound desc = could not find container \"7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2\": container with ID starting with 7ec302270c8e801076ad47ae900c212ced46d5fd86bcc1571cfb752b90ecace2 not found: ID does not exist"
Oct 30 03:44:44.910027 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.910007    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"]
Oct 30 03:44:44.913836 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.913817    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-6d55f47b85-s8fff"]
Oct 30 03:44:44.950813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.950789    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"]
Oct 30 03:44:44.956275 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.956253    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-7fcdfc69c9-8qlx4"]
Oct 30 03:44:44.995150 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.995129    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:44.999680 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:44.999659    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:44:45.158999 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.158966    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e92fff2-f048-484c-92a9-daf34f1e84ac" path="/var/lib/kubelet/pods/1e92fff2-f048-484c-92a9-daf34f1e84ac/volumes"
Oct 30 03:44:45.159341 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.159328    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b838ba-d00d-4095-8d68-823976049cf5" path="/var/lib/kubelet/pods/91b838ba-d00d-4095-8d68-823976049cf5/volumes"
Oct 30 03:44:45.159758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.159746    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafb0bb0-5835-4a9c-982c-14bcb4765b5a" path="/var/lib/kubelet/pods/cafb0bb0-5835-4a9c-982c-14bcb4765b5a/volumes"
Oct 30 03:44:45.491665 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.491629    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo" probeResult="failure" output="Get \"https://10.133.0.72:3101/ready\": dial tcp 10.133.0.72:3101: connect: connection refused"
Oct 30 03:44:45.644573 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.644550    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:44:45.810258 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.810187    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-conf\") pod \"178d490a-506f-4f04-b987-abb5e80b93e5\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") "
Oct 30 03:44:45.810258 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.810221    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-ca-bundle\") pod \"178d490a-506f-4f04-b987-abb5e80b93e5\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") "
Oct 30 03:44:45.810258 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.810250    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178d490a-506f-4f04-b987-abb5e80b93e5-kube-api-access-9j2m6\" (UniqueName: \"kubernetes.io/projected/178d490a-506f-4f04-b987-abb5e80b93e5-kube-api-access-9j2m6\") pod \"178d490a-506f-4f04-b987-abb5e80b93e5\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") "
Oct 30 03:44:45.810499 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.810378    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-tmp-storage\") pod \"178d490a-506f-4f04-b987-abb5e80b93e5\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") "
Oct 30 03:44:45.810499 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.810465    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-compactor-mtls\") pod \"178d490a-506f-4f04-b987-abb5e80b93e5\" (UID: \"178d490a-506f-4f04-b987-abb5e80b93e5\") "
Oct 30 03:44:45.810730 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.810700    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-ca-bundle" pod "178d490a-506f-4f04-b987-abb5e80b93e5" (UID: "178d490a-506f-4f04-b987-abb5e80b93e5"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:45.810792 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.810714    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-conf" pod "178d490a-506f-4f04-b987-abb5e80b93e5" (UID: "178d490a-506f-4f04-b987-abb5e80b93e5"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:45.810792 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.810739    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-tmp-storage" pod "178d490a-506f-4f04-b987-abb5e80b93e5" (UID: "178d490a-506f-4f04-b987-abb5e80b93e5"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:44:45.812387 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.812365    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-compactor-mtls" pod "178d490a-506f-4f04-b987-abb5e80b93e5" (UID: "178d490a-506f-4f04-b987-abb5e80b93e5"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:45.812476 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.812402    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178d490a-506f-4f04-b987-abb5e80b93e5-kube-api-access-9j2m6" pod "178d490a-506f-4f04-b987-abb5e80b93e5" (UID: "178d490a-506f-4f04-b987-abb5e80b93e5"). InnerVolumeSpecName "kube-api-access-9j2m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:45.861184 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.861157    4334 generic.go:356] "Generic (PLEG): container finished" podID="178d490a-506f-4f04-b987-abb5e80b93e5" containerID="456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91" exitCode=0
Oct 30 03:44:45.861635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.861192    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"
Oct 30 03:44:45.861635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.861239    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" event={"ID":"178d490a-506f-4f04-b987-abb5e80b93e5","Type":"ContainerDied","Data":"456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91"}
Oct 30 03:44:45.861635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.861309    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6" event={"ID":"178d490a-506f-4f04-b987-abb5e80b93e5","Type":"ContainerDied","Data":"1fed18b1f5da80e179c05f2c7fe3b41a64eac9f3207e8c6a06fdc34e47eb0afb"}
Oct 30 03:44:45.861635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.861347    4334 scope.go:122] "RemoveContainer" containerID="456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91"
Oct 30 03:44:45.869893 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.869880    4334 scope.go:122] "RemoveContainer" containerID="456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91"
Oct 30 03:44:45.870161 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:44:45.870141    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91\": container with ID starting with 456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91 not found: ID does not exist" containerID="456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91"
Oct 30 03:44:45.870227 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.870168    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91"} err="failed to get container status \"456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91\": rpc error: code = NotFound desc = could not find container \"456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91\": container with ID starting with 456e1e6f0bcc507cc08ddc92f52ecc5f86c36456fc08c218bd0a7b6996100b91 not found: ID does not exist"
Oct 30 03:44:45.911029 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.911004    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:45.911029 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.911025    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:45.911156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.911035    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j2m6\" (UniqueName: \"kubernetes.io/projected/178d490a-506f-4f04-b987-abb5e80b93e5-kube-api-access-9j2m6\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:45.911156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.911048    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-tmp-storage\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:45.911156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.911061    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/178d490a-506f-4f04-b987-abb5e80b93e5-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:44:45.923887 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.923861    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"]
Oct 30 03:44:45.937572 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:45.937542    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7b9db594d8-dvzg6"]
Oct 30 03:44:46.500218 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:46.500184    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" podUID="84c16162-e73b-47b2-8666-54a29cb405ee" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:46.567010 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:46.566973    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" podUID="1ed9f214-cff6-45da-89c4-165615d1a33e" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:47.158528 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:47.158494    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" path="/var/lib/kubelet/pods/178d490a-506f-4f04-b987-abb5e80b93e5/volumes"
Oct 30 03:44:50.520680 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:50.520640    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:50.978305 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:44:50.978269    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:44:56.499550 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:56.499512    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" podUID="84c16162-e73b-47b2-8666-54a29cb405ee" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:44:56.567260 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:44:56.567222    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" podUID="1ed9f214-cff6-45da-89c4-165615d1a33e" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:45:00.521102 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:00.521052    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:45:03.953784 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:03.953740    4334 generic.go:356] "Generic (PLEG): container finished" podID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerID="f04abe4f5d998b38a81a2c4b66a98e52ea8e5bdf357e9c629dfb89ebfad4e9a2" exitCode=0
Oct 30 03:45:03.954232 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:03.953816    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"c8e33605-87f4-48ea-b630-060470e9a8f0","Type":"ContainerDied","Data":"f04abe4f5d998b38a81a2c4b66a98e52ea8e5bdf357e9c629dfb89ebfad4e9a2"}
Oct 30 03:45:04.011804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.011782    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:04.046143 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.046090    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e33605-87f4-48ea-b630-060470e9a8f0-kube-api-access-m8cmp\" (UniqueName: \"kubernetes.io/projected/c8e33605-87f4-48ea-b630-060470e9a8f0-kube-api-access-m8cmp\") pod \"c8e33605-87f4-48ea-b630-060470e9a8f0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") "
Oct 30 03:45:04.046296 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.046203    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"c8e33605-87f4-48ea-b630-060470e9a8f0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") "
Oct 30 03:45:04.046296 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.046268    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ca-bundle\") pod \"c8e33605-87f4-48ea-b630-060470e9a8f0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") "
Oct 30 03:45:04.046384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.046296    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ingester-mtls\") pod \"c8e33605-87f4-48ea-b630-060470e9a8f0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") "
Oct 30 03:45:04.046384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.046319    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-conf\") pod \"c8e33605-87f4-48ea-b630-060470e9a8f0\" (UID: \"c8e33605-87f4-48ea-b630-060470e9a8f0\") "
Oct 30 03:45:04.046719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.046681    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ca-bundle" pod "c8e33605-87f4-48ea-b630-060470e9a8f0" (UID: "c8e33605-87f4-48ea-b630-060470e9a8f0"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:04.046828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.046807    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-conf" pod "c8e33605-87f4-48ea-b630-060470e9a8f0" (UID: "c8e33605-87f4-48ea-b630-060470e9a8f0"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:04.048301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.048275    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e33605-87f4-48ea-b630-060470e9a8f0-kube-api-access-m8cmp" pod "c8e33605-87f4-48ea-b630-060470e9a8f0" (UID: "c8e33605-87f4-48ea-b630-060470e9a8f0"). InnerVolumeSpecName "kube-api-access-m8cmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:04.048379 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.048352    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ingester-mtls" pod "c8e33605-87f4-48ea-b630-060470e9a8f0" (UID: "c8e33605-87f4-48ea-b630-060470e9a8f0"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:04.048865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.048843    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f" pod "c8e33605-87f4-48ea-b630-060470e9a8f0" (UID: "c8e33605-87f4-48ea-b630-060470e9a8f0"). InnerVolumeSpecName "pvc-f5482def-d74f-49f3-bea3-783ed5622e7e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:45:04.147101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.147062    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:04.147101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.147099    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:04.147101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.147110    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8cmp\" (UniqueName: \"kubernetes.io/projected/c8e33605-87f4-48ea-b630-060470e9a8f0-kube-api-access-m8cmp\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:04.147302 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.147137    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:45:04.147302 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.147147    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e33605-87f4-48ea-b630-060470e9a8f0-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:04.157308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.157290    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-f5482def-d74f-49f3-bea3-783ed5622e7e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:45:04.247752 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.247724    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-41cb6fd7406043f5cdbef35d658cd5b6e8fa4caf7266326e58ff992d85d9d51c\""
Oct 30 03:45:04.959793 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.959758    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"c8e33605-87f4-48ea-b630-060470e9a8f0","Type":"ContainerDied","Data":"bcc0994c7dcad9a99e31c1575322b8f15953a9f16c02d18cdcca348f62c8e95a"}
Oct 30 03:45:04.959793 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.959794    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:04.960253 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:04.959815    4334 scope.go:122] "RemoveContainer" containerID="f04abe4f5d998b38a81a2c4b66a98e52ea8e5bdf357e9c629dfb89ebfad4e9a2"
Oct 30 03:45:05.015724 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:05.015699    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:05.028899 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:05.028874    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:05.159386 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:05.159360    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" path="/var/lib/kubelet/pods/c8e33605-87f4-48ea-b630-060470e9a8f0/volumes"
Oct 30 03:45:06.499570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:06.499537    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" podUID="84c16162-e73b-47b2-8666-54a29cb405ee" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:45:06.500076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:06.499657    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:45:06.666412 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:06.666373    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" podUID="1ed9f214-cff6-45da-89c4-165615d1a33e" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:45:06.666668 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:06.666513    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:45:13.450028 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.450002    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:45:13.452873 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.452851    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:45:13.515224 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515197    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-tmp-storage\") pod \"1ed9f214-cff6-45da-89c4-165615d1a33e\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") "
Oct 30 03:45:13.515363 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515241    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-querier-mtls\") pod \"84c16162-e73b-47b2-8666-54a29cb405ee\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") "
Oct 30 03:45:13.515363 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515262    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-ca-bundle\") pod \"84c16162-e73b-47b2-8666-54a29cb405ee\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") "
Oct 30 03:45:13.515363 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515282    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-tmp-storage\") pod \"84c16162-e73b-47b2-8666-54a29cb405ee\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") "
Oct 30 03:45:13.515363 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515313    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-ca-bundle\") pod \"1ed9f214-cff6-45da-89c4-165615d1a33e\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") "
Oct 30 03:45:13.515363 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515333    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-conf\") pod \"1ed9f214-cff6-45da-89c4-165615d1a33e\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") "
Oct 30 03:45:13.515677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515379    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-conf\") pod \"84c16162-e73b-47b2-8666-54a29cb405ee\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") "
Oct 30 03:45:13.515677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515408    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c16162-e73b-47b2-8666-54a29cb405ee-kube-api-access-56pf2\" (UniqueName: \"kubernetes.io/projected/84c16162-e73b-47b2-8666-54a29cb405ee-kube-api-access-56pf2\") pod \"84c16162-e73b-47b2-8666-54a29cb405ee\" (UID: \"84c16162-e73b-47b2-8666-54a29cb405ee\") "
Oct 30 03:45:13.515677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515462    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed9f214-cff6-45da-89c4-165615d1a33e-kube-api-access-jj4b9\" (UniqueName: \"kubernetes.io/projected/1ed9f214-cff6-45da-89c4-165615d1a33e-kube-api-access-jj4b9\") pod \"1ed9f214-cff6-45da-89c4-165615d1a33e\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") "
Oct 30 03:45:13.515677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515508    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-query-frontend-mtls\") pod \"1ed9f214-cff6-45da-89c4-165615d1a33e\" (UID: \"1ed9f214-cff6-45da-89c4-165615d1a33e\") "
Oct 30 03:45:13.515677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515647    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-tmp-storage" pod "1ed9f214-cff6-45da-89c4-165615d1a33e" (UID: "1ed9f214-cff6-45da-89c4-165615d1a33e"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:13.515918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515711    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-tmp-storage" pod "84c16162-e73b-47b2-8666-54a29cb405ee" (UID: "84c16162-e73b-47b2-8666-54a29cb405ee"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:13.515918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515710    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-ca-bundle" pod "84c16162-e73b-47b2-8666-54a29cb405ee" (UID: "84c16162-e73b-47b2-8666-54a29cb405ee"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:13.515918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515823    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-ca-bundle" pod "1ed9f214-cff6-45da-89c4-165615d1a33e" (UID: "1ed9f214-cff6-45da-89c4-165615d1a33e"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:13.515918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515878    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-tmp-storage\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.515918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515901    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.515918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.515917    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-tmp-storage\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.516196 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.516044    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-conf" pod "1ed9f214-cff6-45da-89c4-165615d1a33e" (UID: "1ed9f214-cff6-45da-89c4-165615d1a33e"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:13.516196 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.516082    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-conf" pod "84c16162-e73b-47b2-8666-54a29cb405ee" (UID: "84c16162-e73b-47b2-8666-54a29cb405ee"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:13.517472 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.517427    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-querier-mtls" pod "84c16162-e73b-47b2-8666-54a29cb405ee" (UID: "84c16162-e73b-47b2-8666-54a29cb405ee"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:13.517660 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.517644    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-query-frontend-mtls" pod "1ed9f214-cff6-45da-89c4-165615d1a33e" (UID: "1ed9f214-cff6-45da-89c4-165615d1a33e"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:13.517836 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.517818    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed9f214-cff6-45da-89c4-165615d1a33e-kube-api-access-jj4b9" pod "1ed9f214-cff6-45da-89c4-165615d1a33e" (UID: "1ed9f214-cff6-45da-89c4-165615d1a33e"). InnerVolumeSpecName "kube-api-access-jj4b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:13.517898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.517879    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c16162-e73b-47b2-8666-54a29cb405ee-kube-api-access-56pf2" pod "84c16162-e73b-47b2-8666-54a29cb405ee" (UID: "84c16162-e73b-47b2-8666-54a29cb405ee"). InnerVolumeSpecName "kube-api-access-56pf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:13.617002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.616934    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.617002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.616958    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.617002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.616969    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.617002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.616979    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1ed9f214-cff6-45da-89c4-165615d1a33e-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.617002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.616987    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84c16162-e73b-47b2-8666-54a29cb405ee-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.617002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.616995    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56pf2\" (UniqueName: \"kubernetes.io/projected/84c16162-e73b-47b2-8666-54a29cb405ee-kube-api-access-56pf2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:13.617002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:13.617004    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj4b9\" (UniqueName: \"kubernetes.io/projected/1ed9f214-cff6-45da-89c4-165615d1a33e-kube-api-access-jj4b9\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:14.004224 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.004189    4334 generic.go:356] "Generic (PLEG): container finished" podID="84c16162-e73b-47b2-8666-54a29cb405ee" containerID="8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f" exitCode=137
Oct 30 03:45:14.004398 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.004232    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"
Oct 30 03:45:14.004398 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.004269    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" event={"ID":"84c16162-e73b-47b2-8666-54a29cb405ee","Type":"ContainerDied","Data":"8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f"}
Oct 30 03:45:14.004398 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.004323    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6" event={"ID":"84c16162-e73b-47b2-8666-54a29cb405ee","Type":"ContainerDied","Data":"3032de51e35132fdafc2e62dc3d20ff13a18ec775f127e08282150ca9630fb6e"}
Oct 30 03:45:14.004398 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.004346    4334 scope.go:122] "RemoveContainer" containerID="8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f"
Oct 30 03:45:14.005797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.005759    4334 generic.go:356] "Generic (PLEG): container finished" podID="1ed9f214-cff6-45da-89c4-165615d1a33e" containerID="329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456" exitCode=137
Oct 30 03:45:14.005901 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.005811    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"
Oct 30 03:45:14.005901 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.005862    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" event={"ID":"1ed9f214-cff6-45da-89c4-165615d1a33e","Type":"ContainerDied","Data":"329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456"}
Oct 30 03:45:14.005901 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.005898    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx" event={"ID":"1ed9f214-cff6-45da-89c4-165615d1a33e","Type":"ContainerDied","Data":"aa5bbf3e5c9d40eff50d022f8d2dbbb18317f503bee6d18e9d83fdd2cf23a7a9"}
Oct 30 03:45:14.012732 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.012710    4334 scope.go:122] "RemoveContainer" containerID="8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f"
Oct 30 03:45:14.013034 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:45:14.013014    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f\": container with ID starting with 8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f not found: ID does not exist" containerID="8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f"
Oct 30 03:45:14.013090 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.013043    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f"} err="failed to get container status \"8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f\": rpc error: code = NotFound desc = could not find container \"8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f\": container with ID starting with 8fd6c0c4e02b188d0e7be25126258a417cfac2c7f5e6cb12eb7add6fab5c2e4f not found: ID does not exist"
Oct 30 03:45:14.013090 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.013062    4334 scope.go:122] "RemoveContainer" containerID="329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456"
Oct 30 03:45:14.021035 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.021021    4334 scope.go:122] "RemoveContainer" containerID="329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456"
Oct 30 03:45:14.021400 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:45:14.021374    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456\": container with ID starting with 329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456 not found: ID does not exist" containerID="329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456"
Oct 30 03:45:14.021513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.021404    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456"} err="failed to get container status \"329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456\": rpc error: code = NotFound desc = could not find container \"329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456\": container with ID starting with 329532ab8a85459de1a6fe34ae5245362b018804aaca53f77162ba8116cf8456 not found: ID does not exist"
Oct 30 03:45:14.063210 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.063186    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"]
Oct 30 03:45:14.076403 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.076379    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8497c5f77f-j4gvx"]
Oct 30 03:45:14.113681 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.113653    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"]
Oct 30 03:45:14.117481 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:14.117439    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-579c44f4c4-zldk6"]
Oct 30 03:45:15.158487 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:15.158442    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed9f214-cff6-45da-89c4-165615d1a33e" path="/var/lib/kubelet/pods/1ed9f214-cff6-45da-89c4-165615d1a33e/volumes"
Oct 30 03:45:15.158877 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:15.158832    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c16162-e73b-47b2-8666-54a29cb405ee" path="/var/lib/kubelet/pods/84c16162-e73b-47b2-8666-54a29cb405ee/volumes"
Oct 30 03:45:20.977775 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:45:20.977742    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:45:24.259397 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259359    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259811    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1e92fff2-f048-484c-92a9-daf34f1e84ac" containerName="otc-container"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259827    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e92fff2-f048-484c-92a9-daf34f1e84ac" containerName="otc-container"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259839    4334 cpu_manager.go:414] "containerMap: removing container" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259844    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259854    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1ed9f214-cff6-45da-89c4-165615d1a33e" containerName="tempo"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259860    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed9f214-cff6-45da-89c4-165615d1a33e" containerName="tempo"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259868    4334 cpu_manager.go:414] "containerMap: removing container" podUID="84c16162-e73b-47b2-8666-54a29cb405ee" containerName="tempo"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259872    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c16162-e73b-47b2-8666-54a29cb405ee" containerName="tempo"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259879    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerName="tempo"
Oct 30 03:45:24.259884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259884    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerName="tempo"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259896    4334 cpu_manager.go:414] "containerMap: removing container" podUID="cafb0bb0-5835-4a9c-982c-14bcb4765b5a" containerName="tempo"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259900    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafb0bb0-5835-4a9c-982c-14bcb4765b5a" containerName="tempo"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259911    4334 cpu_manager.go:414] "containerMap: removing container" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259915    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259925    4334 cpu_manager.go:414] "containerMap: removing container" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway-opa"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259930    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway-opa"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259979    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e92fff2-f048-484c-92a9-daf34f1e84ac" containerName="otc-container"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259987    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="178d490a-506f-4f04-b987-abb5e80b93e5" containerName="tempo"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259992    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed9f214-cff6-45da-89c4-165615d1a33e" containerName="tempo"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.259997    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c16162-e73b-47b2-8666-54a29cb405ee" containerName="tempo"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.260002    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e33605-87f4-48ea-b630-060470e9a8f0" containerName="tempo"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.260008    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafb0bb0-5835-4a9c-982c-14bcb4765b5a" containerName="tempo"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.260014    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway"
Oct 30 03:45:24.260434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.260019    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b838ba-d00d-4095-8d68-823976049cf5" containerName="tempo-gateway-opa"
Oct 30 03:45:24.264709 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.264687    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.281509 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.281422    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:45:24.291794 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.291767    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:45:24.291926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.291816    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:45:24.291926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.291829    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:45:24.291926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.291852    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:45:24.294457 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.294424    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\""
Oct 30 03:45:24.394936 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.394900    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.395100 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.394942    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.395100 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.395034    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcp5\" (UniqueName: \"kubernetes.io/projected/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-kube-api-access-pzcp5\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.395100 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.395062    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.495899 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.495872    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.495899 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.495908    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.496103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.495963    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcp5\" (UniqueName: \"kubernetes.io/projected/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-kube-api-access-pzcp5\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.496103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.495990    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.496691 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.496669    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.498268 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.498243    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.498439 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.498419    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.519646 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.519597    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcp5\" (UniqueName: \"kubernetes.io/projected/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-kube-api-access-pzcp5\") pod \"data-science-collector-collector-0\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.579550 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.579523    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:45:24.889805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.889782    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:45:24.891706 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:45:24.891675    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6369803_6b7e_4eb9_b50a_85dfd3897a4c.slice/crio-0ff905290d98fbdd3ca0ba08f580852e750d4828134446c222cf0e8bff260f70 WatchSource:0}: Error finding container 0ff905290d98fbdd3ca0ba08f580852e750d4828134446c222cf0e8bff260f70: Status 404 returned error can't find the container with id 0ff905290d98fbdd3ca0ba08f580852e750d4828134446c222cf0e8bff260f70
Oct 30 03:45:24.893733 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.893702    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:45:24.893817 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:24.893759    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:45:25.057015 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:25.056983    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e6369803-6b7e-4eb9-b50a-85dfd3897a4c","Type":"ContainerStarted","Data":"c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f"}
Oct 30 03:45:25.057180 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:25.057039    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e6369803-6b7e-4eb9-b50a-85dfd3897a4c","Type":"ContainerStarted","Data":"0ff905290d98fbdd3ca0ba08f580852e750d4828134446c222cf0e8bff260f70"}
Oct 30 03:45:26.825108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.825060    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.8250469259999997 podStartE2EDuration="2.825046926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:25.103149237 +0000 UTC m=+3804.109739492" watchObservedRunningTime="2026-10-30 03:45:26.825046926 +0000 UTC m=+3805.831637179"
Oct 30 03:45:26.826164 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.826146    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"]
Oct 30 03:45:26.829290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.829275    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:26.859664 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.859638    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"]
Oct 30 03:45:26.894053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.894026    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\""
Oct 30 03:45:26.894216 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.894141    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\""
Oct 30 03:45:26.898144 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.898118    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:26.901881 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.901863    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:26.913764 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.913744    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:26.913852 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.913785    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:26.913852 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.913804    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:26.913935 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.913925    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:26.913976 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.913945    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g54x\" (UniqueName: \"kubernetes.io/projected/9b71f75f-d0bd-48f3-b771-1fa736db2f51-kube-api-access-6g54x\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:26.914208 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.914191    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\""
Oct 30 03:45:26.914276 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.914203    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-b5l6v\""
Oct 30 03:45:26.941626 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:26.941593    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:27.015007 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.014977    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdld\" (UniqueName: \"kubernetes.io/projected/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-kube-api-access-7pdld\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.015172 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015012    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.015172 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015046    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.015172 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015093    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.015300 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015254    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.015300 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015282    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\""
Oct 30 03:45:27.015399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015318    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.015399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015326    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\""
Oct 30 03:45:27.015399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015370    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.015399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015393    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.015610 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015415    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g54x\" (UniqueName: \"kubernetes.io/projected/9b71f75f-d0bd-48f3-b771-1fa736db2f51-kube-api-access-6g54x\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.015610 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015478    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.015610 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015513    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.016025 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.015992    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.016139 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.016060    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.017919 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.017898    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.031875 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.031850    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"]
Oct 30 03:45:27.035286 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.035271    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.038538 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.038518    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g54x\" (UniqueName: \"kubernetes.io/projected/9b71f75f-d0bd-48f3-b771-1fa736db2f51-kube-api-access-6g54x\") pod \"tempo-data-science-tempostack-distributor-775c964ffc-76nwd\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.069734 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.069705    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"]
Oct 30 03:45:27.104202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.104143    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"]
Oct 30 03:45:27.107349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.107334    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.116038 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116021    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdld\" (UniqueName: \"kubernetes.io/projected/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-kube-api-access-7pdld\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.116115 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116049    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.116115 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116085    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.116207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116120    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.116207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116140    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.116207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116155    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.116475 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116214    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.116632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116607    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgww\" (UniqueName: \"kubernetes.io/projected/847b15ae-f4d9-42e1-b57e-82f03515b7ac-kube-api-access-6fgww\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.116716 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116674    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.116882 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.116862    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.117027 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.117003    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.118076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.117724    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\""
Oct 30 03:45:27.118076 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.117845    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.118302 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.118276    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.120579 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.120555    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.125383 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.125361    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.139942 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.139919    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"]
Oct 30 03:45:27.141201 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.141184    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:27.147703 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.147682    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdld\" (UniqueName: \"kubernetes.io/projected/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-kube-api-access-7pdld\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:27.200061 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.200010    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"]
Oct 30 03:45:27.207398 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.206982    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.216802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.216781    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\""
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218187    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218241    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218500    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218548    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218579    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218650    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgww\" (UniqueName: \"kubernetes.io/projected/847b15ae-f4d9-42e1-b57e-82f03515b7ac-kube-api-access-6fgww\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218676    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218715    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218791    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218822    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpkm\" (UniqueName: \"kubernetes.io/projected/c0509172-bd3b-4f56-b675-dfe66ff31b06-kube-api-access-wcpkm\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218852    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.219145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.218877    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.219740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.219471    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.220855 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.220029    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.220855 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.220282    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.222037 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.222008    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.222806 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.222787    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.229962 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.229942    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"]
Oct 30 03:45:27.241288 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.241260    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgww\" (UniqueName: \"kubernetes.io/projected/847b15ae-f4d9-42e1-b57e-82f03515b7ac-kube-api-access-6fgww\") pod \"tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.283688 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.283658    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\""
Oct 30 03:45:27.320290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320258    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.320427 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320296    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.320427 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320329    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.320427 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320386    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.320427 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320417    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.320609 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320473    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpkm\" (UniqueName: \"kubernetes.io/projected/c0509172-bd3b-4f56-b675-dfe66ff31b06-kube-api-access-wcpkm\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.320609 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320515    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.320744 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320682    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.320744 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320727    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.320838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320772    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.320838 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320785    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.320928 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320855    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.320980 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.320962    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22mzp\" (UniqueName: \"kubernetes.io/projected/36b95fd9-76ee-427d-8137-f41baf0fd614-kube-api-access-22mzp\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.321382 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.321360    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.321547 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.321432    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.323108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.323084    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.323196 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.323120    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.340660 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.340638    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpkm\" (UniqueName: \"kubernetes.io/projected/c0509172-bd3b-4f56-b675-dfe66ff31b06-kube-api-access-wcpkm\") pod \"tempo-data-science-tempostack-querier-7d6858b8c-mmlqc\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.354487 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.354427    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:27.422024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.421990    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.422159 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.422038    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.422159 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.422090    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mzp\" (UniqueName: \"kubernetes.io/projected/36b95fd9-76ee-427d-8137-f41baf0fd614-kube-api-access-22mzp\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.422159 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.422131    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.422323 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.422179    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.422323 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.422206    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.423324 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.423298    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.423489 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.423463    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.423746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.423729    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.425327 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.425309    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:27.425695 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.425672    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.425996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.425974    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.445899 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.445874    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mzp\" (UniqueName: \"kubernetes.io/projected/36b95fd9-76ee-427d-8137-f41baf0fd614-kube-api-access-22mzp\") pod \"tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.457746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.457722    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"]
Oct 30 03:45:27.459179 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:45:27.459156    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b71f75f_d0bd_48f3_b771_1fa736db2f51.slice/crio-9bbd470e01f4127a5b5519c47fd4392bbc15b2bdbfcf4dfdd9468eededb8fabc WatchSource:0}: Error finding container 9bbd470e01f4127a5b5519c47fd4392bbc15b2bdbfcf4dfdd9468eededb8fabc: Status 404 returned error can't find the container with id 9bbd470e01f4127a5b5519c47fd4392bbc15b2bdbfcf4dfdd9468eededb8fabc
Oct 30 03:45:27.520585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.520554    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:27.650064 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.650039    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"]
Oct 30 03:45:27.651915 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:45:27.651888    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847b15ae_f4d9_42e1_b57e_82f03515b7ac.slice/crio-16e01661974f0b5cf2bfa01f6361f304205c5ad894edc2730e49cb083b7e5a36 WatchSource:0}: Error finding container 16e01661974f0b5cf2bfa01f6361f304205c5ad894edc2730e49cb083b7e5a36: Status 404 returned error can't find the container with id 16e01661974f0b5cf2bfa01f6361f304205c5ad894edc2730e49cb083b7e5a36
Oct 30 03:45:27.727208 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.727186    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"]
Oct 30 03:45:27.729258 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:45:27.729234    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0509172_bd3b_4f56_b675_dfe66ff31b06.slice/crio-7c205c8b0cebcc58a2fa4f91f4719dbbfac0bb8f62455ff9a68a0fded28c4172 WatchSource:0}: Error finding container 7c205c8b0cebcc58a2fa4f91f4719dbbfac0bb8f62455ff9a68a0fded28c4172: Status 404 returned error can't find the container with id 7c205c8b0cebcc58a2fa4f91f4719dbbfac0bb8f62455ff9a68a0fded28c4172
Oct 30 03:45:27.789400 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.789374    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:27.806411 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:45:27.806385    4334 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="4cfdb2b1-77ae-43b7-8a29-6629aaad6519"
Oct 30 03:45:27.824343 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:27.824317    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"]
Oct 30 03:45:27.866629 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:45:27.866587    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b95fd9_76ee_427d_8137_f41baf0fd614.slice/crio-330a6741ff6d57ca2df35ae086dee4a1ff20d0bd1f232c65c3e8ac421d9c0348 WatchSource:0}: Error finding container 330a6741ff6d57ca2df35ae086dee4a1ff20d0bd1f232c65c3e8ac421d9c0348: Status 404 returned error can't find the container with id 330a6741ff6d57ca2df35ae086dee4a1ff20d0bd1f232c65c3e8ac421d9c0348
Oct 30 03:45:28.004406 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.004377    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"]
Oct 30 03:45:28.007873 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.007857    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.031459 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.031430    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\""
Oct 30 03:45:28.031696 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.031682    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-pxnpq\""
Oct 30 03:45:28.032915 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.032899    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\""
Oct 30 03:45:28.043667 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.043648    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:45:28.045093 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.045064    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:45:28.045418 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.045400    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\""
Oct 30 03:45:28.048865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.048842    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"]
Oct 30 03:45:28.073840 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.073811    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" event={"ID":"c0509172-bd3b-4f56-b675-dfe66ff31b06","Type":"ContainerStarted","Data":"7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367"}
Oct 30 03:45:28.073966 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.073865    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" event={"ID":"c0509172-bd3b-4f56-b675-dfe66ff31b06","Type":"ContainerStarted","Data":"7c205c8b0cebcc58a2fa4f91f4719dbbfac0bb8f62455ff9a68a0fded28c4172"}
Oct 30 03:45:28.073966 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.073944    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:28.075630 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.075606    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd" event={"ID":"9b71f75f-d0bd-48f3-b771-1fa736db2f51","Type":"ContainerStarted","Data":"399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87"}
Oct 30 03:45:28.075746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.075640    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd" event={"ID":"9b71f75f-d0bd-48f3-b771-1fa736db2f51","Type":"ContainerStarted","Data":"9bbd470e01f4127a5b5519c47fd4392bbc15b2bdbfcf4dfdd9468eededb8fabc"}
Oct 30 03:45:28.075746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.075679    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:28.077378 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.077348    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" event={"ID":"36b95fd9-76ee-427d-8137-f41baf0fd614","Type":"ContainerStarted","Data":"51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4"}
Oct 30 03:45:28.077522 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.077394    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:45:28.077522 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.077417    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" event={"ID":"36b95fd9-76ee-427d-8137-f41baf0fd614","Type":"ContainerStarted","Data":"330a6741ff6d57ca2df35ae086dee4a1ff20d0bd1f232c65c3e8ac421d9c0348"}
Oct 30 03:45:28.079036 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.079017    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:28.079147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.079061    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" event={"ID":"847b15ae-f4d9-42e1-b57e-82f03515b7ac","Type":"ContainerStarted","Data":"02f79e91d3fc7d1cc7c87e742ad795401dd5846e0860eb016602c235db9a8a76"}
Oct 30 03:45:28.079147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.079134    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" event={"ID":"847b15ae-f4d9-42e1-b57e-82f03515b7ac","Type":"ContainerStarted","Data":"16e01661974f0b5cf2bfa01f6361f304205c5ad894edc2730e49cb083b7e5a36"}
Oct 30 03:45:28.085926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.085909    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:28.128956 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.128934    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ca-bundle\") pod \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") "
Oct 30 03:45:28.129070 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.128984    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-conf\") pod \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") "
Oct 30 03:45:28.129131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129069    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-kube-api-access-7pdld\" (UniqueName: \"kubernetes.io/projected/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-kube-api-access-7pdld\") pod \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") "
Oct 30 03:45:28.129131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129104    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-storage-gcs-key\") pod \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") "
Oct 30 03:45:28.129240 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129142    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ingester-mtls\") pod \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\" (UID: \"4cfdb2b1-77ae-43b7-8a29-6629aaad6519\") "
Oct 30 03:45:28.129292 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129272    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.129358 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129312    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ca-bundle" pod "4cfdb2b1-77ae-43b7-8a29-6629aaad6519" (UID: "4cfdb2b1-77ae-43b7-8a29-6629aaad6519"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:28.129442 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129405    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-conf" pod "4cfdb2b1-77ae-43b7-8a29-6629aaad6519" (UID: "4cfdb2b1-77ae-43b7-8a29-6629aaad6519"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:28.129442 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129415    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.129442 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129502    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64xsk\" (UniqueName: \"kubernetes.io/projected/349119c9-2275-4127-959a-5c52f9826e1e-kube-api-access-64xsk\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.129442 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129550    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-rbac\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.129442 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.129728    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.130604 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.130362    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-cabundle\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.130604 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.130577    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tenant\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.130710 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.130654    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.130710 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.130672    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.131620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.131601    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-kube-api-access-7pdld" pod "4cfdb2b1-77ae-43b7-8a29-6629aaad6519" (UID: "4cfdb2b1-77ae-43b7-8a29-6629aaad6519"). InnerVolumeSpecName "kube-api-access-7pdld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:28.131771 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.131753    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ingester-mtls" pod "4cfdb2b1-77ae-43b7-8a29-6629aaad6519" (UID: "4cfdb2b1-77ae-43b7-8a29-6629aaad6519"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:28.131771 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.131764    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-storage-gcs-key" pod "4cfdb2b1-77ae-43b7-8a29-6629aaad6519" (UID: "4cfdb2b1-77ae-43b7-8a29-6629aaad6519"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:28.152225 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.152185    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd" podStartSLOduration=2.152174164 podStartE2EDuration="2.152174164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:28.145907637 +0000 UTC m=+3807.152497891" watchObservedRunningTime="2026-10-30 03:45:28.152174164 +0000 UTC m=+3807.158764420"
Oct 30 03:45:28.202561 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.202522    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" podStartSLOduration=1.202509755 podStartE2EDuration="1.202509755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:28.201199493 +0000 UTC m=+3807.207789747" watchObservedRunningTime="2026-10-30 03:45:28.202509755 +0000 UTC m=+3807.209100008"
Oct 30 03:45:28.231745 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.231719    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.231853 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.231770    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64xsk\" (UniqueName: \"kubernetes.io/projected/349119c9-2275-4127-959a-5c52f9826e1e-kube-api-access-64xsk\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.231853 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.231826    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-rbac\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.231973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.231903    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.231973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.231944    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-cabundle\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.232093 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.232021    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tenant\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.232718 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.232280    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.232718 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.232356    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdld\" (UniqueName: \"kubernetes.io/projected/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-kube-api-access-7pdld\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.232718 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.232374    4334 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-storage-gcs-key\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.232718 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.232391    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/4cfdb2b1-77ae-43b7-8a29-6629aaad6519-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.232984 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.232815    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-rbac\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.232984 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.232836    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-cabundle\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.232984 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.232963    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.234411 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.234387    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.234598 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.234578    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tenant\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.234650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.234578    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.245252 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.245208    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" podStartSLOduration=1.245194141 podStartE2EDuration="1.245194141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:28.24489931 +0000 UTC m=+3807.251489565" watchObservedRunningTime="2026-10-30 03:45:28.245194141 +0000 UTC m=+3807.251784399"
Oct 30 03:45:28.252046 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.252027    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64xsk\" (UniqueName: \"kubernetes.io/projected/349119c9-2275-4127-959a-5c52f9826e1e-kube-api-access-64xsk\") pod \"tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.286129 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.286090    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" podStartSLOduration=1.286080924 podStartE2EDuration="1.286080924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:28.285313577 +0000 UTC m=+3807.291903831" watchObservedRunningTime="2026-10-30 03:45:28.286080924 +0000 UTC m=+3807.292671178"
Oct 30 03:45:28.322591 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.322570    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:28.644768 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:28.644745    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"]
Oct 30 03:45:28.645349 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:45:28.645323    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349119c9_2275_4127_959a_5c52f9826e1e.slice/crio-d0e795f71cc828e2a91648240a8bb3002b05b0897ef28a014d97d83a3be661b4 WatchSource:0}: Error finding container d0e795f71cc828e2a91648240a8bb3002b05b0897ef28a014d97d83a3be661b4: Status 404 returned error can't find the container with id d0e795f71cc828e2a91648240a8bb3002b05b0897ef28a014d97d83a3be661b4
Oct 30 03:45:29.085465 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.085410    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.085465 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.085429    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" event={"ID":"349119c9-2275-4127-959a-5c52f9826e1e","Type":"ContainerStarted","Data":"0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4"}
Oct 30 03:45:29.085895 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.085514    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" event={"ID":"349119c9-2275-4127-959a-5c52f9826e1e","Type":"ContainerStarted","Data":"d0e795f71cc828e2a91648240a8bb3002b05b0897ef28a014d97d83a3be661b4"}
Oct 30 03:45:29.085895 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.085890    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:29.165648 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.165616    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:29.168351 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.168320    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:29.208249 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.208219    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:29.213066 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.213048    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.230501 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.230475    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:29.231335 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.231310    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\""
Oct 30 03:45:29.342894 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.342868    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.342983 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.342918    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxr7m\" (UniqueName: \"kubernetes.io/projected/b40f30ad-3443-4791-beaa-85eebdfcecba-kube-api-access-rxr7m\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.343019 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.342982    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.343113 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.343091    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.343155 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.343133    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.443843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.443788    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.443843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.443833    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxr7m\" (UniqueName: \"kubernetes.io/projected/b40f30ad-3443-4791-beaa-85eebdfcecba-kube-api-access-rxr7m\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.443979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.443854    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.443979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.443901    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.443979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.443919    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.444623 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.444592    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.444899 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.444874    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.446329 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.446308    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.446442 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.446360    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:29.464108 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:29.464088    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxr7m\" (UniqueName: \"kubernetes.io/projected/b40f30ad-3443-4791-beaa-85eebdfcecba-kube-api-access-rxr7m\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:30.092419 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.092374    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" event={"ID":"349119c9-2275-4127-959a-5c52f9826e1e","Type":"ContainerStarted","Data":"d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15"}
Oct 30 03:45:30.092909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.092650    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:30.092909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.092696    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:30.094045 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.094014    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.82:8083/ready\": dial tcp 10.133.0.82:8083: connect: connection refused"
Oct 30 03:45:30.140968 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.140923    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" podStartSLOduration=3.140908982 podStartE2EDuration="3.140908982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:30.139177124 +0000 UTC m=+3809.145767377" watchObservedRunningTime="2026-10-30 03:45:30.140908982 +0000 UTC m=+3809.147499262"
Oct 30 03:45:30.654323 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.654292    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:30.668570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.668540    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:30.697565 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.697535    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:30.755304 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.755279    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:30.755437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.755387    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:30.770482 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.770441    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") DevicePath \"csi-41cb6fd7406043f5cdbef35d658cd5b6e8fa4caf7266326e58ff992d85d9d51c\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:30.871577 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.871553    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/239011135fcbd13577b023b82456a4d813e5268c79848b272c29ac9f9cbb7c39/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:30.898585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:30.898561    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:31.024744 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:31.024677    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:31.099934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:31.099905    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:45:31.159899 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:31.159865    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cfdb2b1-77ae-43b7-8a29-6629aaad6519" path="/var/lib/kubelet/pods/4cfdb2b1-77ae-43b7-8a29-6629aaad6519/volumes"
Oct 30 03:45:31.335072 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:31.334997    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:45:31.337350 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:45:31.337324    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40f30ad_3443_4791_beaa_85eebdfcecba.slice/crio-d1a503b6eff39b407aa5351c1e59969477bc6fbc6eb1a4cd546f304c22c47d7e WatchSource:0}: Error finding container d1a503b6eff39b407aa5351c1e59969477bc6fbc6eb1a4cd546f304c22c47d7e: Status 404 returned error can't find the container with id d1a503b6eff39b407aa5351c1e59969477bc6fbc6eb1a4cd546f304c22c47d7e
Oct 30 03:45:32.105405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:32.105374    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"b40f30ad-3443-4791-beaa-85eebdfcecba","Type":"ContainerStarted","Data":"3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980"}
Oct 30 03:45:32.105904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:32.105422    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"b40f30ad-3443-4791-beaa-85eebdfcecba","Type":"ContainerStarted","Data":"d1a503b6eff39b407aa5351c1e59969477bc6fbc6eb1a4cd546f304c22c47d7e"}
Oct 30 03:45:32.105904 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:32.105458    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:45:32.147613 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:32.147568    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=3.14755604 podStartE2EDuration="3.14755604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:32.146043674 +0000 UTC m=+3811.152633930" watchObservedRunningTime="2026-10-30 03:45:32.14755604 +0000 UTC m=+3811.154146294"
Oct 30 03:45:49.091816 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:49.091725    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:49.092392 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:49.092305    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:45:49.092511 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:49.092414    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:45:50.166603 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:50.166566    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:45:50.977772 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:45:50.977738    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:45:53.116160 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:53.116126    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:45:53.637525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:53.637495    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"]
Oct 30 03:45:53.637748 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:53.637727    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" podUID="c0509172-bd3b-4f56-b675-dfe66ff31b06" containerName="tempo" containerID="cri-o://7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367" gracePeriod=30
Oct 30 03:45:53.692939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:53.692907    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"]
Oct 30 03:45:53.693116 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:53.693090    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd" podUID="9b71f75f-d0bd-48f3-b771-1fa736db2f51" containerName="tempo" containerID="cri-o://399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87" gracePeriod=30
Oct 30 03:45:53.778053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:53.778016    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"]
Oct 30 03:45:53.778242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:53.778222    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" podUID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" containerName="tempo" containerID="cri-o://02f79e91d3fc7d1cc7c87e742ad795401dd5846e0860eb016602c235db9a8a76" gracePeriod=30
Oct 30 03:45:54.839663 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.839640    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:54.944103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944036    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-ca-bundle\") pod \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") "
Oct 30 03:45:54.944103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944072    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-tmp-storage\") pod \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") "
Oct 30 03:45:54.944103 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944093    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-conf\") pod \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") "
Oct 30 03:45:54.944347 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944163    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-distributor-mtls\") pod \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") "
Oct 30 03:45:54.944347 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944229    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b71f75f-d0bd-48f3-b771-1fa736db2f51-kube-api-access-6g54x\" (UniqueName: \"kubernetes.io/projected/9b71f75f-d0bd-48f3-b771-1fa736db2f51-kube-api-access-6g54x\") pod \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\" (UID: \"9b71f75f-d0bd-48f3-b771-1fa736db2f51\") "
Oct 30 03:45:54.944487 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944357    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-tmp-storage" pod "9b71f75f-d0bd-48f3-b771-1fa736db2f51" (UID: "9b71f75f-d0bd-48f3-b771-1fa736db2f51"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:54.944487 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944471    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-ca-bundle" pod "9b71f75f-d0bd-48f3-b771-1fa736db2f51" (UID: "9b71f75f-d0bd-48f3-b771-1fa736db2f51"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:54.944610 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944527    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-tmp-storage\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:54.944610 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.944588    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-conf" pod "9b71f75f-d0bd-48f3-b771-1fa736db2f51" (UID: "9b71f75f-d0bd-48f3-b771-1fa736db2f51"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:54.946385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.946356    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-distributor-mtls" pod "9b71f75f-d0bd-48f3-b771-1fa736db2f51" (UID: "9b71f75f-d0bd-48f3-b771-1fa736db2f51"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:54.946385 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:54.946370    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b71f75f-d0bd-48f3-b771-1fa736db2f51-kube-api-access-6g54x" pod "9b71f75f-d0bd-48f3-b771-1fa736db2f51" (UID: "9b71f75f-d0bd-48f3-b771-1fa736db2f51"). InnerVolumeSpecName "kube-api-access-6g54x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:55.045390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.045361    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g54x\" (UniqueName: \"kubernetes.io/projected/9b71f75f-d0bd-48f3-b771-1fa736db2f51-kube-api-access-6g54x\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:55.045390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.045387    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:55.045556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.045397    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:55.045556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.045405    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/9b71f75f-d0bd-48f3-b771-1fa736db2f51-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:45:55.222187 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.222122    4334 generic.go:356] "Generic (PLEG): container finished" podID="9b71f75f-d0bd-48f3-b771-1fa736db2f51" containerID="399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87" exitCode=0
Oct 30 03:45:55.222187 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.222162    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"
Oct 30 03:45:55.222352 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.222208    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd" event={"ID":"9b71f75f-d0bd-48f3-b771-1fa736db2f51","Type":"ContainerDied","Data":"399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87"}
Oct 30 03:45:55.222352 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.222263    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd" event={"ID":"9b71f75f-d0bd-48f3-b771-1fa736db2f51","Type":"ContainerDied","Data":"9bbd470e01f4127a5b5519c47fd4392bbc15b2bdbfcf4dfdd9468eededb8fabc"}
Oct 30 03:45:55.222352 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.222287    4334 scope.go:122] "RemoveContainer" containerID="399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87"
Oct 30 03:45:55.230721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.230700    4334 scope.go:122] "RemoveContainer" containerID="399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87"
Oct 30 03:45:55.230988 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:45:55.230970    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87\": container with ID starting with 399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87 not found: ID does not exist" containerID="399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87"
Oct 30 03:45:55.231051 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.230999    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87"} err="failed to get container status \"399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87\": rpc error: code = NotFound desc = could not find container \"399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87\": container with ID starting with 399f48ec3bd0a1cbe1970e3a578e103c7913c100db1d3d669c1025835abaef87 not found: ID does not exist"
Oct 30 03:45:55.274883 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.274851    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"]
Oct 30 03:45:55.287952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:55.287925    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-775c964ffc-76nwd"]
Oct 30 03:45:57.159184 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:57.159143    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b71f75f-d0bd-48f3-b771-1fa736db2f51" path="/var/lib/kubelet/pods/9b71f75f-d0bd-48f3-b771-1fa736db2f51/volumes"
Oct 30 03:45:59.090089 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:59.090042    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:45:59.090587 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:45:59.090564    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" podUID="c0509172-bd3b-4f56-b675-dfe66ff31b06" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:00.466024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:00.465977    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" podUID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:03.115585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:03.115542    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:09.090315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:09.090274    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" podUID="c0509172-bd3b-4f56-b675-dfe66ff31b06" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:09.090775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:09.090656    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:10.365039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:10.365004    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" podUID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:13.114731 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:13.114704    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:46:19.089926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:19.089882    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:19.090367 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:19.090342    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" podUID="c0509172-bd3b-4f56-b675-dfe66ff31b06" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:19.090467 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:19.090438    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:46:20.265243 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:20.265195    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" podUID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:20.265804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:20.265296    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:46:20.977758 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:46:20.977718    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:46:23.795228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.795205    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:46:23.806086 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:46:23.806058    4334 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0509172_bd3b_4f56_b675_dfe66ff31b06.slice/crio-conmon-7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0509172_bd3b_4f56_b675_dfe66ff31b06.slice/crio-7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:46:23.987102 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987069    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-tmp-storage\") pod \"c0509172-bd3b-4f56-b675-dfe66ff31b06\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") "
Oct 30 03:46:23.987239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987114    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-conf\") pod \"c0509172-bd3b-4f56-b675-dfe66ff31b06\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") "
Oct 30 03:46:23.987239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987146    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-storage-gcs-key\") pod \"c0509172-bd3b-4f56-b675-dfe66ff31b06\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") "
Oct 30 03:46:23.987239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987163    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0509172-bd3b-4f56-b675-dfe66ff31b06-kube-api-access-wcpkm\" (UniqueName: \"kubernetes.io/projected/c0509172-bd3b-4f56-b675-dfe66ff31b06-kube-api-access-wcpkm\") pod \"c0509172-bd3b-4f56-b675-dfe66ff31b06\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") "
Oct 30 03:46:23.987239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987190    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-ca-bundle\") pod \"c0509172-bd3b-4f56-b675-dfe66ff31b06\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") "
Oct 30 03:46:23.987239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987225    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-querier-mtls\") pod \"c0509172-bd3b-4f56-b675-dfe66ff31b06\" (UID: \"c0509172-bd3b-4f56-b675-dfe66ff31b06\") "
Oct 30 03:46:23.987594 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987408    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-tmp-storage" pod "c0509172-bd3b-4f56-b675-dfe66ff31b06" (UID: "c0509172-bd3b-4f56-b675-dfe66ff31b06"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:23.987693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987663    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-ca-bundle" pod "c0509172-bd3b-4f56-b675-dfe66ff31b06" (UID: "c0509172-bd3b-4f56-b675-dfe66ff31b06"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:23.987748 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.987695    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-conf" pod "c0509172-bd3b-4f56-b675-dfe66ff31b06" (UID: "c0509172-bd3b-4f56-b675-dfe66ff31b06"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:23.989309 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.989282    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-storage-gcs-key" pod "c0509172-bd3b-4f56-b675-dfe66ff31b06" (UID: "c0509172-bd3b-4f56-b675-dfe66ff31b06"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:23.989415 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.989351    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0509172-bd3b-4f56-b675-dfe66ff31b06-kube-api-access-wcpkm" pod "c0509172-bd3b-4f56-b675-dfe66ff31b06" (UID: "c0509172-bd3b-4f56-b675-dfe66ff31b06"). InnerVolumeSpecName "kube-api-access-wcpkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:23.989415 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:23.989359    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-querier-mtls" pod "c0509172-bd3b-4f56-b675-dfe66ff31b06" (UID: "c0509172-bd3b-4f56-b675-dfe66ff31b06"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:24.088779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.088755    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-tmp-storage\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.088779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.088778    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.088922 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.088788    4334 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-storage-gcs-key\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.088922 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.088797    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpkm\" (UniqueName: \"kubernetes.io/projected/c0509172-bd3b-4f56-b675-dfe66ff31b06-kube-api-access-wcpkm\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.088922 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.088807    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.088922 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.088816    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/c0509172-bd3b-4f56-b675-dfe66ff31b06-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.360187 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.360154    4334 generic.go:356] "Generic (PLEG): container finished" podID="c0509172-bd3b-4f56-b675-dfe66ff31b06" containerID="7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367" exitCode=137
Oct 30 03:46:24.360314 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.360202    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"
Oct 30 03:46:24.360314 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.360243    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" event={"ID":"c0509172-bd3b-4f56-b675-dfe66ff31b06","Type":"ContainerDied","Data":"7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367"}
Oct 30 03:46:24.360314 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.360308    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc" event={"ID":"c0509172-bd3b-4f56-b675-dfe66ff31b06","Type":"ContainerDied","Data":"7c205c8b0cebcc58a2fa4f91f4719dbbfac0bb8f62455ff9a68a0fded28c4172"}
Oct 30 03:46:24.360480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.360342    4334 scope.go:122] "RemoveContainer" containerID="7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367"
Oct 30 03:46:24.362860 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.362839    4334 generic.go:356] "Generic (PLEG): container finished" podID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" containerID="02f79e91d3fc7d1cc7c87e742ad795401dd5846e0860eb016602c235db9a8a76" exitCode=137
Oct 30 03:46:24.362984 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.362878    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" event={"ID":"847b15ae-f4d9-42e1-b57e-82f03515b7ac","Type":"ContainerDied","Data":"02f79e91d3fc7d1cc7c87e742ad795401dd5846e0860eb016602c235db9a8a76"}
Oct 30 03:46:24.373186 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.373161    4334 scope.go:122] "RemoveContainer" containerID="7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367"
Oct 30 03:46:24.373497 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:46:24.373465    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367\": container with ID starting with 7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367 not found: ID does not exist" containerID="7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367"
Oct 30 03:46:24.373603 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.373507    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367"} err="failed to get container status \"7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367\": rpc error: code = NotFound desc = could not find container \"7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367\": container with ID starting with 7e3e3602294cfb1af6d1ef323447a606dc27af00eb75823254d9f89b7a929367 not found: ID does not exist"
Oct 30 03:46:24.406713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.406692    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:46:24.414214 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.414193    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"]
Oct 30 03:46:24.428091 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.428070    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7d6858b8c-mmlqc"]
Oct 30 03:46:24.592127 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592041    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-ca-bundle\") pod \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") "
Oct 30 03:46:24.592127 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592081    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-storage-gcs-key\") pod \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") "
Oct 30 03:46:24.592127 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592105    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847b15ae-f4d9-42e1-b57e-82f03515b7ac-kube-api-access-6fgww\" (UniqueName: \"kubernetes.io/projected/847b15ae-f4d9-42e1-b57e-82f03515b7ac-kube-api-access-6fgww\") pod \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") "
Oct 30 03:46:24.592408 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592154    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-conf\") pod \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") "
Oct 30 03:46:24.592408 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592203    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-tmp-storage\") pod \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") "
Oct 30 03:46:24.592408 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592236    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-query-frontend-mtls\") pod \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\" (UID: \"847b15ae-f4d9-42e1-b57e-82f03515b7ac\") "
Oct 30 03:46:24.592614 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592582    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-tmp-storage" pod "847b15ae-f4d9-42e1-b57e-82f03515b7ac" (UID: "847b15ae-f4d9-42e1-b57e-82f03515b7ac"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:24.592685 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592616    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-ca-bundle" pod "847b15ae-f4d9-42e1-b57e-82f03515b7ac" (UID: "847b15ae-f4d9-42e1-b57e-82f03515b7ac"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:24.592685 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.592656    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-conf" pod "847b15ae-f4d9-42e1-b57e-82f03515b7ac" (UID: "847b15ae-f4d9-42e1-b57e-82f03515b7ac"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:24.594277 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.594248    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847b15ae-f4d9-42e1-b57e-82f03515b7ac-kube-api-access-6fgww" pod "847b15ae-f4d9-42e1-b57e-82f03515b7ac" (UID: "847b15ae-f4d9-42e1-b57e-82f03515b7ac"). InnerVolumeSpecName "kube-api-access-6fgww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:24.594383 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.594286    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-query-frontend-mtls" pod "847b15ae-f4d9-42e1-b57e-82f03515b7ac" (UID: "847b15ae-f4d9-42e1-b57e-82f03515b7ac"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:24.594383 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.594302    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-storage-gcs-key" pod "847b15ae-f4d9-42e1-b57e-82f03515b7ac" (UID: "847b15ae-f4d9-42e1-b57e-82f03515b7ac"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:24.692929 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.692900    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.692929 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.692924    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.693071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.692934    4334 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/847b15ae-f4d9-42e1-b57e-82f03515b7ac-storage-gcs-key\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.693071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.692943    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgww\" (UniqueName: \"kubernetes.io/projected/847b15ae-f4d9-42e1-b57e-82f03515b7ac-kube-api-access-6fgww\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.693071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.692952    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:24.693071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:24.692960    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/847b15ae-f4d9-42e1-b57e-82f03515b7ac-tempo-tmp-storage\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:25.159638 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:25.159605    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0509172-bd3b-4f56-b675-dfe66ff31b06" path="/var/lib/kubelet/pods/c0509172-bd3b-4f56-b675-dfe66ff31b06/volumes"
Oct 30 03:46:25.370559 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:25.370528    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"
Oct 30 03:46:25.370707 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:25.370530    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw" event={"ID":"847b15ae-f4d9-42e1-b57e-82f03515b7ac","Type":"ContainerDied","Data":"16e01661974f0b5cf2bfa01f6361f304205c5ad894edc2730e49cb083b7e5a36"}
Oct 30 03:46:25.370707 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:25.370661    4334 scope.go:122] "RemoveContainer" containerID="02f79e91d3fc7d1cc7c87e742ad795401dd5846e0860eb016602c235db9a8a76"
Oct 30 03:46:25.424474 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:25.424395    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"]
Oct 30 03:46:25.439109 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:25.439077    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-75757f7fcb-7jmsw"]
Oct 30 03:46:27.158953 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:27.158922    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" path="/var/lib/kubelet/pods/847b15ae-f4d9-42e1-b57e-82f03515b7ac/volumes"
Oct 30 03:46:29.089817 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:29.089782    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:33.640682 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:33.640649    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"]
Oct 30 03:46:33.641092 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:33.640866    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo" containerID="cri-o://51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4" gracePeriod=30
Oct 30 03:46:33.644671 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:33.644637    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:36.592497 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.592473    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:46:36.671331 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.671299    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-conf\") pod \"36b95fd9-76ee-427d-8137-f41baf0fd614\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") "
Oct 30 03:46:36.671513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.671347    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b95fd9-76ee-427d-8137-f41baf0fd614-kube-api-access-22mzp\" (UniqueName: \"kubernetes.io/projected/36b95fd9-76ee-427d-8137-f41baf0fd614-kube-api-access-22mzp\") pod \"36b95fd9-76ee-427d-8137-f41baf0fd614\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") "
Oct 30 03:46:36.671513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.671386    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-tmp-storage\") pod \"36b95fd9-76ee-427d-8137-f41baf0fd614\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") "
Oct 30 03:46:36.671513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.671414    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-compactor-mtls\") pod \"36b95fd9-76ee-427d-8137-f41baf0fd614\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") "
Oct 30 03:46:36.671513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.671470    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-ca-bundle\") pod \"36b95fd9-76ee-427d-8137-f41baf0fd614\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") "
Oct 30 03:46:36.671857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.671831    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-conf" pod "36b95fd9-76ee-427d-8137-f41baf0fd614" (UID: "36b95fd9-76ee-427d-8137-f41baf0fd614"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:36.671939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.671849    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-tmp-storage" pod "36b95fd9-76ee-427d-8137-f41baf0fd614" (UID: "36b95fd9-76ee-427d-8137-f41baf0fd614"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:36.671939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.671888    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-ca-bundle" pod "36b95fd9-76ee-427d-8137-f41baf0fd614" (UID: "36b95fd9-76ee-427d-8137-f41baf0fd614"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:36.673584 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.673563    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b95fd9-76ee-427d-8137-f41baf0fd614-kube-api-access-22mzp" pod "36b95fd9-76ee-427d-8137-f41baf0fd614" (UID: "36b95fd9-76ee-427d-8137-f41baf0fd614"). InnerVolumeSpecName "kube-api-access-22mzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:36.673668 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.673650    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-compactor-mtls" pod "36b95fd9-76ee-427d-8137-f41baf0fd614" (UID: "36b95fd9-76ee-427d-8137-f41baf0fd614"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:36.772628 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.772604    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-storage-gcs-key\") pod \"36b95fd9-76ee-427d-8137-f41baf0fd614\" (UID: \"36b95fd9-76ee-427d-8137-f41baf0fd614\") "
Oct 30 03:46:36.772808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.772795    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22mzp\" (UniqueName: \"kubernetes.io/projected/36b95fd9-76ee-427d-8137-f41baf0fd614-kube-api-access-22mzp\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:36.772857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.772810    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-tmp-storage\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:36.772857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.772819    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:36.772857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.772829    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:36.772857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.772838    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36b95fd9-76ee-427d-8137-f41baf0fd614-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:36.774560 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.774537    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-storage-gcs-key" pod "36b95fd9-76ee-427d-8137-f41baf0fd614" (UID: "36b95fd9-76ee-427d-8137-f41baf0fd614"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:36.873440 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:36.873408    4334 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36b95fd9-76ee-427d-8137-f41baf0fd614-storage-gcs-key\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:37.430918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.430835    4334 generic.go:356] "Generic (PLEG): container finished" podID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerID="51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4" exitCode=0
Oct 30 03:46:37.430918 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.430889    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"
Oct 30 03:46:37.431095 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.430918    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" event={"ID":"36b95fd9-76ee-427d-8137-f41baf0fd614","Type":"ContainerDied","Data":"51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4"}
Oct 30 03:46:37.431095 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.430975    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc" event={"ID":"36b95fd9-76ee-427d-8137-f41baf0fd614","Type":"ContainerDied","Data":"330a6741ff6d57ca2df35ae086dee4a1ff20d0bd1f232c65c3e8ac421d9c0348"}
Oct 30 03:46:37.431095 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.430997    4334 scope.go:122] "RemoveContainer" containerID="51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4"
Oct 30 03:46:37.439806 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.439788    4334 scope.go:122] "RemoveContainer" containerID="51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4"
Oct 30 03:46:37.440105 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:46:37.440087    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4\": container with ID starting with 51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4 not found: ID does not exist" containerID="51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4"
Oct 30 03:46:37.440163 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.440116    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4"} err="failed to get container status \"51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4\": rpc error: code = NotFound desc = could not find container \"51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4\": container with ID starting with 51fadff674c4bdb0b4f9ab787670ed3634cd9751955eac5c7df55151235902d4 not found: ID does not exist"
Oct 30 03:46:37.486242 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.486211    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"]
Oct 30 03:46:37.499297 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:37.499272    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-5cc99c5df9-5hwcc"]
Oct 30 03:46:38.944343 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:38.944313    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:46:38.944906 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:38.944493    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="e6369803-6b7e-4eb9-b50a-85dfd3897a4c" containerName="otc-container" containerID="cri-o://c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f" gracePeriod=30
Oct 30 03:46:39.160436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.160397    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" path="/var/lib/kubelet/pods/36b95fd9-76ee-427d-8137-f41baf0fd614/volumes"
Oct 30 03:46:39.183935 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.183913    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.187299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.187275    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-monitoring-tls-certs\") pod \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") "
Oct 30 03:46:39.187351 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.187317    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otlp-tls-certs\") pod \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") "
Oct 30 03:46:39.187351 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.187346    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-kube-api-access-pzcp5\" (UniqueName: \"kubernetes.io/projected/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-kube-api-access-pzcp5\") pod \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") "
Oct 30 03:46:39.187423 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.187391    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otc-internal\") pod \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\" (UID: \"e6369803-6b7e-4eb9-b50a-85dfd3897a4c\") "
Oct 30 03:46:39.187783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.187760    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otc-internal" pod "e6369803-6b7e-4eb9-b50a-85dfd3897a4c" (UID: "e6369803-6b7e-4eb9-b50a-85dfd3897a4c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:39.189411 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.189382    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-kube-api-access-pzcp5" pod "e6369803-6b7e-4eb9-b50a-85dfd3897a4c" (UID: "e6369803-6b7e-4eb9-b50a-85dfd3897a4c"). InnerVolumeSpecName "kube-api-access-pzcp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:39.189531 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.189420    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otlp-tls-certs" pod "e6369803-6b7e-4eb9-b50a-85dfd3897a4c" (UID: "e6369803-6b7e-4eb9-b50a-85dfd3897a4c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:39.190578 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.190562    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-monitoring-tls-certs" pod "e6369803-6b7e-4eb9-b50a-85dfd3897a4c" (UID: "e6369803-6b7e-4eb9-b50a-85dfd3897a4c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:39.288444 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.288366    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:39.288444 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.288393    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:39.288444 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.288403    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzcp5\" (UniqueName: \"kubernetes.io/projected/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-kube-api-access-pzcp5\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:39.288444 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.288412    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6369803-6b7e-4eb9-b50a-85dfd3897a4c-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:46:39.442396 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.442358    4334 generic.go:356] "Generic (PLEG): container finished" podID="e6369803-6b7e-4eb9-b50a-85dfd3897a4c" containerID="c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f" exitCode=0
Oct 30 03:46:39.442572 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.442423    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.442572 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.442440    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e6369803-6b7e-4eb9-b50a-85dfd3897a4c","Type":"ContainerDied","Data":"c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f"}
Oct 30 03:46:39.442572 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.442526    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e6369803-6b7e-4eb9-b50a-85dfd3897a4c","Type":"ContainerDied","Data":"0ff905290d98fbdd3ca0ba08f580852e750d4828134446c222cf0e8bff260f70"}
Oct 30 03:46:39.442572 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.442550    4334 scope.go:122] "RemoveContainer" containerID="c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f"
Oct 30 03:46:39.452118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.452096    4334 scope.go:122] "RemoveContainer" containerID="c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f"
Oct 30 03:46:39.452368 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:46:39.452349    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f\": container with ID starting with c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f not found: ID does not exist" containerID="c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f"
Oct 30 03:46:39.452439 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.452375    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f"} err="failed to get container status \"c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f\": rpc error: code = NotFound desc = could not find container \"c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f\": container with ID starting with c4486ddfded0c3f33b4a60b6b7eeab79e87dfdfdce8e95fd6e0db2aa40ebc71f not found: ID does not exist"
Oct 30 03:46:39.498643 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.498616    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:46:39.502249 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.502230    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:46:39.538288 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538264    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:46:39.538632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538595    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9b71f75f-d0bd-48f3-b771-1fa736db2f51" containerName="tempo"
Oct 30 03:46:39.538632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538609    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b71f75f-d0bd-48f3-b771-1fa736db2f51" containerName="tempo"
Oct 30 03:46:39.538632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538622    4334 cpu_manager.go:414] "containerMap: removing container" podUID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" containerName="tempo"
Oct 30 03:46:39.538632 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538629    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" containerName="tempo"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538638    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e6369803-6b7e-4eb9-b50a-85dfd3897a4c" containerName="otc-container"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538643    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6369803-6b7e-4eb9-b50a-85dfd3897a4c" containerName="otc-container"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538654    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c0509172-bd3b-4f56-b675-dfe66ff31b06" containerName="tempo"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538661    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0509172-bd3b-4f56-b675-dfe66ff31b06" containerName="tempo"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538673    4334 cpu_manager.go:414] "containerMap: removing container" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538679    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538736    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="847b15ae-f4d9-42e1-b57e-82f03515b7ac" containerName="tempo"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538748    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6369803-6b7e-4eb9-b50a-85dfd3897a4c" containerName="otc-container"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538754    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0509172-bd3b-4f56-b675-dfe66ff31b06" containerName="tempo"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538761    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b95fd9-76ee-427d-8137-f41baf0fd614" containerName="tempo"
Oct 30 03:46:39.538788 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.538769    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b71f75f-d0bd-48f3-b771-1fa736db2f51" containerName="tempo"
Oct 30 03:46:39.543058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.543042    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.559192 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.559168    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\""
Oct 30 03:46:39.559570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.559552    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:46:39.559719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.559705    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:46:39.559793 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.559782    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7h6s\""
Oct 30 03:46:39.559865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.559844    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:46:39.591097 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.591059    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/69d5a127-71fd-4e98-91a0-2f832c2ae028-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.591228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.591100    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/69d5a127-71fd-4e98-91a0-2f832c2ae028-kube-api-access-h29kl\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.591228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.591209    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.591335 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.591250    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.692038 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.692000    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.692234 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.692045    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.692234 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.692084    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/69d5a127-71fd-4e98-91a0-2f832c2ae028-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.692234 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.692099    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/69d5a127-71fd-4e98-91a0-2f832c2ae028-kube-api-access-h29kl\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.692780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.692746    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/69d5a127-71fd-4e98-91a0-2f832c2ae028-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.694636 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.694612    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.694732 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.694617    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.712003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.711977    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/69d5a127-71fd-4e98-91a0-2f832c2ae028-kube-api-access-h29kl\") pod \"data-science-collector-collector-0\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:39.855241 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:39.855192    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:46:40.151597 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:40.151569    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:46:40.153333 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:46:40.153307    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d5a127_71fd_4e98_91a0_2f832c2ae028.slice/crio-808651d689f5bb9253541531a4798746383950d15637db483d0e08c7b2864b27 WatchSource:0}: Error finding container 808651d689f5bb9253541531a4798746383950d15637db483d0e08c7b2864b27: Status 404 returned error can't find the container with id 808651d689f5bb9253541531a4798746383950d15637db483d0e08c7b2864b27
Oct 30 03:46:40.155525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:40.155491    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:46:40.155609 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:40.155546    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:46:40.451320 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:40.451279    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"69d5a127-71fd-4e98-91a0-2f832c2ae028","Type":"ContainerStarted","Data":"9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614"}
Oct 30 03:46:40.451488 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:40.451354    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"69d5a127-71fd-4e98-91a0-2f832c2ae028","Type":"ContainerStarted","Data":"808651d689f5bb9253541531a4798746383950d15637db483d0e08c7b2864b27"}
Oct 30 03:46:40.495399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:40.495339    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.495319077 podStartE2EDuration="1.495319077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:40.494976285 +0000 UTC m=+3879.501566538" watchObservedRunningTime="2026-10-30 03:46:40.495319077 +0000 UTC m=+3879.501909333"
Oct 30 03:46:41.159028 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:46:41.158995    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6369803-6b7e-4eb9-b50a-85dfd3897a4c" path="/var/lib/kubelet/pods/e6369803-6b7e-4eb9-b50a-85dfd3897a4c/volumes"
Oct 30 03:46:50.978072 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:46:50.978035    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:47:10.528432 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.528400    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:10.528963 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.528702    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="69d5a127-71fd-4e98-91a0-2f832c2ae028" containerName="otc-container" containerID="cri-o://9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614" gracePeriod=30
Oct 30 03:47:10.790185 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.790128    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:10.932032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.932002    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d5a127-71fd-4e98-91a0-2f832c2ae028-kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/69d5a127-71fd-4e98-91a0-2f832c2ae028-kube-api-access-h29kl\") pod \"69d5a127-71fd-4e98-91a0-2f832c2ae028\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") "
Oct 30 03:47:10.932192 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.932081    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-monitoring-tls-certs\") pod \"69d5a127-71fd-4e98-91a0-2f832c2ae028\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") "
Oct 30 03:47:10.932192 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.932102    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d5a127-71fd-4e98-91a0-2f832c2ae028-otc-internal\" (UniqueName: \"kubernetes.io/configmap/69d5a127-71fd-4e98-91a0-2f832c2ae028-otc-internal\") pod \"69d5a127-71fd-4e98-91a0-2f832c2ae028\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") "
Oct 30 03:47:10.932192 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.932148    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-otlp-tls-certs\") pod \"69d5a127-71fd-4e98-91a0-2f832c2ae028\" (UID: \"69d5a127-71fd-4e98-91a0-2f832c2ae028\") "
Oct 30 03:47:10.932520 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.932485    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d5a127-71fd-4e98-91a0-2f832c2ae028-otc-internal" pod "69d5a127-71fd-4e98-91a0-2f832c2ae028" (UID: "69d5a127-71fd-4e98-91a0-2f832c2ae028"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:10.934129 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.934107    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-monitoring-tls-certs" pod "69d5a127-71fd-4e98-91a0-2f832c2ae028" (UID: "69d5a127-71fd-4e98-91a0-2f832c2ae028"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:10.934210 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.934165    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d5a127-71fd-4e98-91a0-2f832c2ae028-kube-api-access-h29kl" pod "69d5a127-71fd-4e98-91a0-2f832c2ae028" (UID: "69d5a127-71fd-4e98-91a0-2f832c2ae028"). InnerVolumeSpecName "kube-api-access-h29kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:10.934258 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:10.934240    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-otlp-tls-certs" pod "69d5a127-71fd-4e98-91a0-2f832c2ae028" (UID: "69d5a127-71fd-4e98-91a0-2f832c2ae028"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:11.033583 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.033550    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.033583 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.033582    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/69d5a127-71fd-4e98-91a0-2f832c2ae028-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.033828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.033596    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5a127-71fd-4e98-91a0-2f832c2ae028-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.033828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.033608    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h29kl\" (UniqueName: \"kubernetes.io/projected/69d5a127-71fd-4e98-91a0-2f832c2ae028-kube-api-access-h29kl\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:11.113005 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.112907    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:47:11.113201 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.113155    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerName="tempo" containerID="cri-o://3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980" gracePeriod=30
Oct 30 03:47:11.163239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.163201    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"]
Oct 30 03:47:11.163534 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.163484    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway" containerID="cri-o://0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4" gracePeriod=30
Oct 30 03:47:11.163615 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.163594    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway-opa" containerID="cri-o://d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15" gracePeriod=30
Oct 30 03:47:11.593802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.593766    4334 generic.go:356] "Generic (PLEG): container finished" podID="69d5a127-71fd-4e98-91a0-2f832c2ae028" containerID="9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614" exitCode=0
Oct 30 03:47:11.594217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.593826    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:11.594217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.593853    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"69d5a127-71fd-4e98-91a0-2f832c2ae028","Type":"ContainerDied","Data":"9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614"}
Oct 30 03:47:11.594217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.593953    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"69d5a127-71fd-4e98-91a0-2f832c2ae028","Type":"ContainerDied","Data":"808651d689f5bb9253541531a4798746383950d15637db483d0e08c7b2864b27"}
Oct 30 03:47:11.594217 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.593985    4334 scope.go:122] "RemoveContainer" containerID="9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614"
Oct 30 03:47:11.596051 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.596025    4334 generic.go:356] "Generic (PLEG): container finished" podID="349119c9-2275-4127-959a-5c52f9826e1e" containerID="d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15" exitCode=0
Oct 30 03:47:11.596193 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.596123    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" event={"ID":"349119c9-2275-4127-959a-5c52f9826e1e","Type":"ContainerDied","Data":"d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15"}
Oct 30 03:47:11.604551 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.604535    4334 scope.go:122] "RemoveContainer" containerID="9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614"
Oct 30 03:47:11.604822 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:11.604805    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614\": container with ID starting with 9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614 not found: ID does not exist" containerID="9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614"
Oct 30 03:47:11.604870 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.604830    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614"} err="failed to get container status \"9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614\": rpc error: code = NotFound desc = could not find container \"9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614\": container with ID starting with 9fa673e95bd845361d7e4171e00af66821f4830469b1780d7fb3a79d4db9c614 not found: ID does not exist"
Oct 30 03:47:11.650971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.650945    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:11.657254 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:11.657233    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:12.413189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.413167    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:47:12.546086 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546011    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-rbac\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-rbac\") pod \"349119c9-2275-4127-959a-5c52f9826e1e\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") "
Oct 30 03:47:12.546086 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546048    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-ca-bundle\") pod \"349119c9-2275-4127-959a-5c52f9826e1e\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") "
Oct 30 03:47:12.546291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546101    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-gateway-mtls\") pod \"349119c9-2275-4127-959a-5c52f9826e1e\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") "
Oct 30 03:47:12.546291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546203    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tenant\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tenant\") pod \"349119c9-2275-4127-959a-5c52f9826e1e\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") "
Oct 30 03:47:12.546291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546249    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-serving-certs\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-serving-certs\") pod \"349119c9-2275-4127-959a-5c52f9826e1e\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") "
Oct 30 03:47:12.546291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546275    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349119c9-2275-4127-959a-5c52f9826e1e-kube-api-access-64xsk\" (UniqueName: \"kubernetes.io/projected/349119c9-2275-4127-959a-5c52f9826e1e-kube-api-access-64xsk\") pod \"349119c9-2275-4127-959a-5c52f9826e1e\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") "
Oct 30 03:47:12.546530 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546297    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-cabundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-cabundle\") pod \"349119c9-2275-4127-959a-5c52f9826e1e\" (UID: \"349119c9-2275-4127-959a-5c52f9826e1e\") "
Oct 30 03:47:12.546530 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546426    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-rbac" pod "349119c9-2275-4127-959a-5c52f9826e1e" (UID: "349119c9-2275-4127-959a-5c52f9826e1e"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:12.546530 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546494    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-ca-bundle" pod "349119c9-2275-4127-959a-5c52f9826e1e" (UID: "349119c9-2275-4127-959a-5c52f9826e1e"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:12.546694 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546591    4334 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.546746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546719    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.546804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.546778    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-cabundle" pod "349119c9-2275-4127-959a-5c52f9826e1e" (UID: "349119c9-2275-4127-959a-5c52f9826e1e"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:12.548306 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.548281    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-serving-certs" pod "349119c9-2275-4127-959a-5c52f9826e1e" (UID: "349119c9-2275-4127-959a-5c52f9826e1e"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:12.548435 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.548343    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-gateway-mtls" pod "349119c9-2275-4127-959a-5c52f9826e1e" (UID: "349119c9-2275-4127-959a-5c52f9826e1e"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:12.548435 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.548359    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349119c9-2275-4127-959a-5c52f9826e1e-kube-api-access-64xsk" pod "349119c9-2275-4127-959a-5c52f9826e1e" (UID: "349119c9-2275-4127-959a-5c52f9826e1e"). InnerVolumeSpecName "kube-api-access-64xsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:12.551926 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.551904    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tenant" pod "349119c9-2275-4127-959a-5c52f9826e1e" (UID: "349119c9-2275-4127-959a-5c52f9826e1e"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:12.602733 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.602706    4334 generic.go:356] "Generic (PLEG): container finished" podID="349119c9-2275-4127-959a-5c52f9826e1e" containerID="0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4" exitCode=0
Oct 30 03:47:12.603061 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.602763    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"
Oct 30 03:47:12.603061 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.602785    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" event={"ID":"349119c9-2275-4127-959a-5c52f9826e1e","Type":"ContainerDied","Data":"0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4"}
Oct 30 03:47:12.603061 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.602838    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq" event={"ID":"349119c9-2275-4127-959a-5c52f9826e1e","Type":"ContainerDied","Data":"d0e795f71cc828e2a91648240a8bb3002b05b0897ef28a014d97d83a3be661b4"}
Oct 30 03:47:12.603061 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.602863    4334 scope.go:122] "RemoveContainer" containerID="d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15"
Oct 30 03:47:12.611228 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.611211    4334 scope.go:122] "RemoveContainer" containerID="0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4"
Oct 30 03:47:12.618837 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.618821    4334 scope.go:122] "RemoveContainer" containerID="d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15"
Oct 30 03:47:12.619078 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:12.619061    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15\": container with ID starting with d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15 not found: ID does not exist" containerID="d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15"
Oct 30 03:47:12.619123 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.619087    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15"} err="failed to get container status \"d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15\": rpc error: code = NotFound desc = could not find container \"d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15\": container with ID starting with d3585181c294cbe135076e89d7d5ffb899210c05cc5af7f02c16bf0fac73ef15 not found: ID does not exist"
Oct 30 03:47:12.619123 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.619102    4334 scope.go:122] "RemoveContainer" containerID="0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4"
Oct 30 03:47:12.619312 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:12.619297    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4\": container with ID starting with 0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4 not found: ID does not exist" containerID="0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4"
Oct 30 03:47:12.619349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.619316    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4"} err="failed to get container status \"0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4\": rpc error: code = NotFound desc = could not find container \"0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4\": container with ID starting with 0bdeed3b802c5c610cd812ed8645bee4a5e1806cdf275dfbb34e05feacdfe2d4 not found: ID does not exist"
Oct 30 03:47:12.647234 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.647213    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.647234 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.647231    4334 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-tenant\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.647349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.647241    4334 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/349119c9-2275-4127-959a-5c52f9826e1e-serving-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.647349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.647253    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64xsk\" (UniqueName: \"kubernetes.io/projected/349119c9-2275-4127-959a-5c52f9826e1e-kube-api-access-64xsk\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.647349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.647261    4334 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/349119c9-2275-4127-959a-5c52f9826e1e-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:12.657374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.657349    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"]
Oct 30 03:47:12.660308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:12.660287    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5f65db84bd-h7vcq"]
Oct 30 03:47:13.115133 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:13.115092    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:47:13.158920 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:13.158890    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349119c9-2275-4127-959a-5c52f9826e1e" path="/var/lib/kubelet/pods/349119c9-2275-4127-959a-5c52f9826e1e/volumes"
Oct 30 03:47:13.159304 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:13.159291    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d5a127-71fd-4e98-91a0-2f832c2ae028" path="/var/lib/kubelet/pods/69d5a127-71fd-4e98-91a0-2f832c2ae028/volumes"
Oct 30 03:47:15.619419 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619387    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:15.619833 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619818    4334 cpu_manager.go:414] "containerMap: removing container" podUID="69d5a127-71fd-4e98-91a0-2f832c2ae028" containerName="otc-container"
Oct 30 03:47:15.619877 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619835    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d5a127-71fd-4e98-91a0-2f832c2ae028" containerName="otc-container"
Oct 30 03:47:15.619877 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619856    4334 cpu_manager.go:414] "containerMap: removing container" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway"
Oct 30 03:47:15.619877 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619865    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway"
Oct 30 03:47:15.619973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619878    4334 cpu_manager.go:414] "containerMap: removing container" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway-opa"
Oct 30 03:47:15.619973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619887    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway-opa"
Oct 30 03:47:15.620038 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619973    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d5a127-71fd-4e98-91a0-2f832c2ae028" containerName="otc-container"
Oct 30 03:47:15.620038 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619987    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway"
Oct 30 03:47:15.620038 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.619995    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="349119c9-2275-4127-959a-5c52f9826e1e" containerName="tempo-gateway-opa"
Oct 30 03:47:15.624557 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.624511    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.648612 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.645472    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:15.652641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.652588    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\""
Oct 30 03:47:15.653002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.652981    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:47:15.653207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.653167    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-ptc4w\""
Oct 30 03:47:15.653207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.653203    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:47:15.771283 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.771253    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nms6k\" (UniqueName: \"kubernetes.io/projected/88e9b991-653f-49c2-8628-9e198a610f1e-kube-api-access-nms6k\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.771430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.771288    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.771430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.771329    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e9b991-653f-49c2-8628-9e198a610f1e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.771430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.771416    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.872616 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.872535    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e9b991-653f-49c2-8628-9e198a610f1e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.872616 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.872576    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.872807 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.872649    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nms6k\" (UniqueName: \"kubernetes.io/projected/88e9b991-653f-49c2-8628-9e198a610f1e-kube-api-access-nms6k\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.872807 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.872676    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.873318 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.873287    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e9b991-653f-49c2-8628-9e198a610f1e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.875107 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.875083    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.875192 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.875105    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.892236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.892206    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nms6k\" (UniqueName: \"kubernetes.io/projected/88e9b991-653f-49c2-8628-9e198a610f1e-kube-api-access-nms6k\") pod \"data-science-collector-collector-0\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:15.956189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:15.956165    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:16.251781 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:16.251759    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:16.254222 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:47:16.254194    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9b991_653f_49c2_8628_9e198a610f1e.slice/crio-1a1c6e7a76d18883320830b981ad14b9ac436d9a47a3616d5b8d8a6996680bb4 WatchSource:0}: Error finding container 1a1c6e7a76d18883320830b981ad14b9ac436d9a47a3616d5b8d8a6996680bb4: Status 404 returned error can't find the container with id 1a1c6e7a76d18883320830b981ad14b9ac436d9a47a3616d5b8d8a6996680bb4
Oct 30 03:47:16.256332 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:16.256303    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:47:16.256398 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:16.256360    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:47:16.625797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:16.625709    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"88e9b991-653f-49c2-8628-9e198a610f1e","Type":"ContainerStarted","Data":"976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2"}
Oct 30 03:47:16.625797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:16.625759    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"88e9b991-653f-49c2-8628-9e198a610f1e","Type":"ContainerStarted","Data":"1a1c6e7a76d18883320830b981ad14b9ac436d9a47a3616d5b8d8a6996680bb4"}
Oct 30 03:47:16.668460 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:16.668411    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.668394402 podStartE2EDuration="1.668394402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:16.668024973 +0000 UTC m=+3915.674615227" watchObservedRunningTime="2026-10-30 03:47:16.668394402 +0000 UTC m=+3915.674984657"
Oct 30 03:47:17.257518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.257485    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:47:17.261107 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.261090    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.276689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.276664    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:47:17.276973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.276951    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:47:17.277094 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.276999    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:47:17.277094 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.277004    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:47:17.280253 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.280229    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:47:17.280658 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.280622    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:47:17.292259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.292237    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:47:17.349239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.349213    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:47:17.349369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.349301    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-9nlf4\""
Oct 30 03:47:17.387529 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.387503    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.387641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.387533    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.387641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.387566    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgsp\" (UniqueName: \"kubernetes.io/projected/c52de779-72cd-40b2-9fa7-e1da40599b2e-kube-api-access-fzgsp\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.387727 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.387692    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.387767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.387731    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.387802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.387760    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.387843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.387822    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.387882 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.387846    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.489152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.489128    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.489255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.489161    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.489255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.489183    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.489255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.489214    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.489255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.489246    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.489405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.489287    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.489484 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.489434    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.489554 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.489531    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgsp\" (UniqueName: \"kubernetes.io/projected/c52de779-72cd-40b2-9fa7-e1da40599b2e-kube-api-access-fzgsp\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.490034 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.490010    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.490245 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.490226    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.490342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.490319    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.490705 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.490684    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.491766 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.491747    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.491988 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.491968    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.492048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.491982    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:17.509380 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:17.509324    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgsp\" (UniqueName: \"kubernetes.io/projected/c52de779-72cd-40b2-9fa7-e1da40599b2e-kube-api-access-fzgsp\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:20.979384 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:20.979344    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:47:21.520992 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:21.520965    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:21.551584 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:21.551558    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:21.622159 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:21.622131    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:21.622281 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:21.622239    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:21.637315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:21.637288    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:21.739762 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:21.739730    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0f4ace3e6f98968798883337bc6fba9989453020fd940336937c79983332b77b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:21.771347 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:21.771278    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:22.079689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.079615    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:22.420127 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.420101    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:47:22.422607 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:47:22.422580    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52de779_72cd_40b2_9fa7_e1da40599b2e.slice/crio-f3b9c0bc1ec287e2d7215b31b3054d4af54374246ba9839b47b7348d22340f08 WatchSource:0}: Error finding container f3b9c0bc1ec287e2d7215b31b3054d4af54374246ba9839b47b7348d22340f08: Status 404 returned error can't find the container with id f3b9c0bc1ec287e2d7215b31b3054d4af54374246ba9839b47b7348d22340f08
Oct 30 03:47:22.654553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.654515    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"c52de779-72cd-40b2-9fa7-e1da40599b2e","Type":"ContainerStarted","Data":"6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539"}
Oct 30 03:47:22.654693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.654580    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"c52de779-72cd-40b2-9fa7-e1da40599b2e","Type":"ContainerStarted","Data":"f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b"}
Oct 30 03:47:22.654693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.654600    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"c52de779-72cd-40b2-9fa7-e1da40599b2e","Type":"ContainerStarted","Data":"0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417"}
Oct 30 03:47:22.654693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.654619    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:22.654693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.654638    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:22.654693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.654651    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"c52de779-72cd-40b2-9fa7-e1da40599b2e","Type":"ContainerStarted","Data":"f3b9c0bc1ec287e2d7215b31b3054d4af54374246ba9839b47b7348d22340f08"}
Oct 30 03:47:22.657040 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.657021    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:22.662745 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.662726    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:22.708606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:22.708536    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=5.708523615 podStartE2EDuration="5.708523615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:22.698866501 +0000 UTC m=+3921.705456755" watchObservedRunningTime="2026-10-30 03:47:22.708523615 +0000 UTC m=+3921.715113868"
Oct 30 03:47:23.114396 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:23.114317    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:47:23.658756 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:23.658731    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:31.855767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:31.855743    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:47:32.003601 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.003570    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-storage-gcs-key\") pod \"b40f30ad-3443-4791-beaa-85eebdfcecba\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") "
Oct 30 03:47:32.003601 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.003603    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40f30ad-3443-4791-beaa-85eebdfcecba-kube-api-access-rxr7m\" (UniqueName: \"kubernetes.io/projected/b40f30ad-3443-4791-beaa-85eebdfcecba-kube-api-access-rxr7m\") pod \"b40f30ad-3443-4791-beaa-85eebdfcecba\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") "
Oct 30 03:47:32.003805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.003685    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") pod \"b40f30ad-3443-4791-beaa-85eebdfcecba\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") "
Oct 30 03:47:32.003805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.003707    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ingester-mtls\") pod \"b40f30ad-3443-4791-beaa-85eebdfcecba\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") "
Oct 30 03:47:32.003805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.003739    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ca-bundle\") pod \"b40f30ad-3443-4791-beaa-85eebdfcecba\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") "
Oct 30 03:47:32.003958 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.003820    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-conf\") pod \"b40f30ad-3443-4791-beaa-85eebdfcecba\" (UID: \"b40f30ad-3443-4791-beaa-85eebdfcecba\") "
Oct 30 03:47:32.004211 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.004180    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ca-bundle" pod "b40f30ad-3443-4791-beaa-85eebdfcecba" (UID: "b40f30ad-3443-4791-beaa-85eebdfcecba"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:32.004343 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.004205    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-conf" pod "b40f30ad-3443-4791-beaa-85eebdfcecba" (UID: "b40f30ad-3443-4791-beaa-85eebdfcecba"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:32.005998 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.005965    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ingester-mtls" pod "b40f30ad-3443-4791-beaa-85eebdfcecba" (UID: "b40f30ad-3443-4791-beaa-85eebdfcecba"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:32.006357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.006333    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-storage-gcs-key" pod "b40f30ad-3443-4791-beaa-85eebdfcecba" (UID: "b40f30ad-3443-4791-beaa-85eebdfcecba"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:32.006357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.006344    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40f30ad-3443-4791-beaa-85eebdfcecba-kube-api-access-rxr7m" pod "b40f30ad-3443-4791-beaa-85eebdfcecba" (UID: "b40f30ad-3443-4791-beaa-85eebdfcecba"). InnerVolumeSpecName "kube-api-access-rxr7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:32.006585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.006564    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f" pod "b40f30ad-3443-4791-beaa-85eebdfcecba" (UID: "b40f30ad-3443-4791-beaa-85eebdfcecba"). InnerVolumeSpecName "pvc-f5482def-d74f-49f3-bea3-783ed5622e7e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:47:32.105356 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.105328    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:32.105356 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.105352    4334 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-storage-gcs-key\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:32.105545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.105362    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxr7m\" (UniqueName: \"kubernetes.io/projected/b40f30ad-3443-4791-beaa-85eebdfcecba-kube-api-access-rxr7m\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:32.105545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.105390    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:47:32.105545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.105400    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:32.105545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.105410    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f30ad-3443-4791-beaa-85eebdfcecba-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:32.113630 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.113615    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-f5482def-d74f-49f3-bea3-783ed5622e7e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:47:32.206772 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.206747    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-f5482def-d74f-49f3-bea3-783ed5622e7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d8ee3bd365a0ef8f\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-41cb6fd7406043f5cdbef35d658cd5b6e8fa4caf7266326e58ff992d85d9d51c\""
Oct 30 03:47:32.701641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.701606    4334 generic.go:356] "Generic (PLEG): container finished" podID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerID="3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980" exitCode=0
Oct 30 03:47:32.701827 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.701658    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"b40f30ad-3443-4791-beaa-85eebdfcecba","Type":"ContainerDied","Data":"3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980"}
Oct 30 03:47:32.701827 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.701702    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"b40f30ad-3443-4791-beaa-85eebdfcecba","Type":"ContainerDied","Data":"d1a503b6eff39b407aa5351c1e59969477bc6fbc6eb1a4cd546f304c22c47d7e"}
Oct 30 03:47:32.701827 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.701725    4334 scope.go:122] "RemoveContainer" containerID="3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980"
Oct 30 03:47:32.701827 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.701662    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:47:32.710641 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.710623    4334 scope.go:122] "RemoveContainer" containerID="3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980"
Oct 30 03:47:32.710932 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:32.710914    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980\": container with ID starting with 3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980 not found: ID does not exist" containerID="3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980"
Oct 30 03:47:32.710988 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.710940    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980"} err="failed to get container status \"3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980\": rpc error: code = NotFound desc = could not find container \"3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980\": container with ID starting with 3d17abf9bd9f3f4d1a44fab1d2fe760edc51fadd5106ddf28b3406d12a411980 not found: ID does not exist"
Oct 30 03:47:32.755473 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.755437    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:47:32.768746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:32.768728    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:47:33.159250 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.159221    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" path="/var/lib/kubelet/pods/b40f30ad-3443-4791-beaa-85eebdfcecba/volumes"
Oct 30 03:47:33.323965 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.323932    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:33.324189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.324165    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="88e9b991-653f-49c2-8628-9e198a610f1e" containerName="otc-container" containerID="cri-o://976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2" gracePeriod=30
Oct 30 03:47:33.570375 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.570352    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:33.708218 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.708132    4334 generic.go:356] "Generic (PLEG): container finished" podID="88e9b991-653f-49c2-8628-9e198a610f1e" containerID="976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2" exitCode=0
Oct 30 03:47:33.708218 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.708179    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:33.708473 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.708221    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"88e9b991-653f-49c2-8628-9e198a610f1e","Type":"ContainerDied","Data":"976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2"}
Oct 30 03:47:33.708473 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.708316    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"88e9b991-653f-49c2-8628-9e198a610f1e","Type":"ContainerDied","Data":"1a1c6e7a76d18883320830b981ad14b9ac436d9a47a3616d5b8d8a6996680bb4"}
Oct 30 03:47:33.708473 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.708348    4334 scope.go:122] "RemoveContainer" containerID="976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2"
Oct 30 03:47:33.716535 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.716517    4334 scope.go:122] "RemoveContainer" containerID="976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2"
Oct 30 03:47:33.716793 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:33.716776    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2\": container with ID starting with 976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2 not found: ID does not exist" containerID="976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2"
Oct 30 03:47:33.716841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.716800    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2"} err="failed to get container status \"976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2\": rpc error: code = NotFound desc = could not find container \"976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2\": container with ID starting with 976b1989ced61a9cc36be287da4043fadb005c13b7bca391309e64b17135e9d2 not found: ID does not exist"
Oct 30 03:47:33.719036 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.719017    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e9b991-653f-49c2-8628-9e198a610f1e-kube-api-access-nms6k\" (UniqueName: \"kubernetes.io/projected/88e9b991-653f-49c2-8628-9e198a610f1e-kube-api-access-nms6k\") pod \"88e9b991-653f-49c2-8628-9e198a610f1e\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") "
Oct 30 03:47:33.719101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.719054    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e9b991-653f-49c2-8628-9e198a610f1e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e9b991-653f-49c2-8628-9e198a610f1e-otc-internal\") pod \"88e9b991-653f-49c2-8628-9e198a610f1e\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") "
Oct 30 03:47:33.719101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.719082    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-monitoring-tls-certs\") pod \"88e9b991-653f-49c2-8628-9e198a610f1e\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") "
Oct 30 03:47:33.719166 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.719156    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-otlp-tls-certs\") pod \"88e9b991-653f-49c2-8628-9e198a610f1e\" (UID: \"88e9b991-653f-49c2-8628-9e198a610f1e\") "
Oct 30 03:47:33.719405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.719383    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e9b991-653f-49c2-8628-9e198a610f1e-otc-internal" pod "88e9b991-653f-49c2-8628-9e198a610f1e" (UID: "88e9b991-653f-49c2-8628-9e198a610f1e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:33.721046 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.721017    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e9b991-653f-49c2-8628-9e198a610f1e-kube-api-access-nms6k" pod "88e9b991-653f-49c2-8628-9e198a610f1e" (UID: "88e9b991-653f-49c2-8628-9e198a610f1e"). InnerVolumeSpecName "kube-api-access-nms6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:33.721139 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.721079    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-otlp-tls-certs" pod "88e9b991-653f-49c2-8628-9e198a610f1e" (UID: "88e9b991-653f-49c2-8628-9e198a610f1e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:33.721183 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.721153    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-monitoring-tls-certs" pod "88e9b991-653f-49c2-8628-9e198a610f1e" (UID: "88e9b991-653f-49c2-8628-9e198a610f1e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:33.800051 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.800008    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:47:33.800381 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.800345    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo-gateway-opa" containerID="cri-o://6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539" gracePeriod=30
Oct 30 03:47:33.800518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.800378    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo-gateway" containerID="cri-o://f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b" gracePeriod=30
Oct 30 03:47:33.800518 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.800345    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo" containerID="cri-o://0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417" gracePeriod=30
Oct 30 03:47:33.819768 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.819747    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.819768 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.819770    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nms6k\" (UniqueName: \"kubernetes.io/projected/88e9b991-653f-49c2-8628-9e198a610f1e-kube-api-access-nms6k\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.819946 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.819780    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e9b991-653f-49c2-8628-9e198a610f1e-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:33.819946 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:33.819788    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e9b991-653f-49c2-8628-9e198a610f1e-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:34.055519 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:34.055495    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:34.061656 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:34.061631    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:34.715397 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:34.715368    4334 generic.go:356] "Generic (PLEG): container finished" podID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerID="6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539" exitCode=0
Oct 30 03:47:34.715397 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:34.715391    4334 generic.go:356] "Generic (PLEG): container finished" podID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerID="0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417" exitCode=0
Oct 30 03:47:34.715845 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:34.715458    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"c52de779-72cd-40b2-9fa7-e1da40599b2e","Type":"ContainerDied","Data":"6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539"}
Oct 30 03:47:34.715845 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:34.715512    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"c52de779-72cd-40b2-9fa7-e1da40599b2e","Type":"ContainerDied","Data":"0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417"}
Oct 30 03:47:35.042013 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.041991    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:35.129733 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.129704    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.129909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.129823    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.129909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.129869    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-tenants\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.129909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.129901    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-mtls\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.130082 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.129952    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52de779-72cd-40b2-9fa7-e1da40599b2e-kube-api-access-fzgsp\" (UniqueName: \"kubernetes.io/projected/c52de779-72cd-40b2-9fa7-e1da40599b2e-kube-api-access-fzgsp\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.130082 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.129993    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-conf\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.130082 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.130036    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-ca-bundle\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.130233 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.130077    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-serving-cabundle" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:35.130233 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.130111    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-rbac\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.130233 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.130164    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"c52de779-72cd-40b2-9fa7-e1da40599b2e\" (UID: \"c52de779-72cd-40b2-9fa7-e1da40599b2e\") "
Oct 30 03:47:35.130517 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.130495    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.130631 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.130516    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-conf" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:35.130631 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.130536    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-ca-bundle" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:35.130855 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.130810    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-rbac" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:35.132660 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.132613    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-mtls" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:35.132796 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.132767    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:35.133098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.133073    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-tenants" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:35.133098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.133084    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:47:35.133218 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.133117    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52de779-72cd-40b2-9fa7-e1da40599b2e-kube-api-access-fzgsp" pod "c52de779-72cd-40b2-9fa7-e1da40599b2e" (UID: "c52de779-72cd-40b2-9fa7-e1da40599b2e"). InnerVolumeSpecName "kube-api-access-fzgsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:35.158360 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.158329    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e9b991-653f-49c2-8628-9e198a610f1e" path="/var/lib/kubelet/pods/88e9b991-653f-49c2-8628-9e198a610f1e/volumes"
Oct 30 03:47:35.231032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.230970    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:47:35.231032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.230997    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.231032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.231009    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.231032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.231021    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgsp\" (UniqueName: \"kubernetes.io/projected/c52de779-72cd-40b2-9fa7-e1da40599b2e-kube-api-access-fzgsp\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.231236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.231035    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-conf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.231236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.231048    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.231236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.231061    4334 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/c52de779-72cd-40b2-9fa7-e1da40599b2e-gateway-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.231236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.231075    4334 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/c52de779-72cd-40b2-9fa7-e1da40599b2e-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.239301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.239281    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:47:35.331579 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.331554    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-16b14853-04bd-45f5-b4ee-559e23aa5d42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00679032e7e957505\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0102c1ee4ed4997d6c4c60b9658e3ceebdee69cbfcdd953945a3f70394f8bd09\""
Oct 30 03:47:35.722721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.722690    4334 generic.go:356] "Generic (PLEG): container finished" podID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerID="f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b" exitCode=0
Oct 30 03:47:35.723121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.722745    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:47:35.723121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.722778    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"c52de779-72cd-40b2-9fa7-e1da40599b2e","Type":"ContainerDied","Data":"f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b"}
Oct 30 03:47:35.723121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.722834    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"c52de779-72cd-40b2-9fa7-e1da40599b2e","Type":"ContainerDied","Data":"f3b9c0bc1ec287e2d7215b31b3054d4af54374246ba9839b47b7348d22340f08"}
Oct 30 03:47:35.723121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.722858    4334 scope.go:122] "RemoveContainer" containerID="6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539"
Oct 30 03:47:35.730885 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.730868    4334 scope.go:122] "RemoveContainer" containerID="f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b"
Oct 30 03:47:35.738419 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.738403    4334 scope.go:122] "RemoveContainer" containerID="0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417"
Oct 30 03:47:35.745762 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.745746    4334 scope.go:122] "RemoveContainer" containerID="6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539"
Oct 30 03:47:35.746025 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:35.746005    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539\": container with ID starting with 6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539 not found: ID does not exist" containerID="6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539"
Oct 30 03:47:35.746074 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.746035    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539"} err="failed to get container status \"6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539\": rpc error: code = NotFound desc = could not find container \"6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539\": container with ID starting with 6d30deb8a17b811232870d59cc88b9ba525cd7aeeedebf5ac4a67855f3fbd539 not found: ID does not exist"
Oct 30 03:47:35.746074 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.746052    4334 scope.go:122] "RemoveContainer" containerID="f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b"
Oct 30 03:47:35.746256 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:35.746238    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b\": container with ID starting with f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b not found: ID does not exist" containerID="f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b"
Oct 30 03:47:35.746294 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.746263    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b"} err="failed to get container status \"f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b\": rpc error: code = NotFound desc = could not find container \"f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b\": container with ID starting with f2e088cd58152a720882f8f1744b2b8bafa920a588ad1d0babc7ab220e9fbe6b not found: ID does not exist"
Oct 30 03:47:35.746294 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.746276    4334 scope.go:122] "RemoveContainer" containerID="0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417"
Oct 30 03:47:35.746483 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:35.746469    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417\": container with ID starting with 0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417 not found: ID does not exist" containerID="0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417"
Oct 30 03:47:35.746533 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.746485    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417"} err="failed to get container status \"0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417\": rpc error: code = NotFound desc = could not find container \"0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417\": container with ID starting with 0b2b82a66f94359e9a8cfb34921b3997ff0d8f37d3483e6c4faf8a0e8a523417 not found: ID does not exist"
Oct 30 03:47:35.790942 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.790918    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:47:35.804147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:35.804121    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:47:37.157962 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.157929    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" path="/var/lib/kubelet/pods/c52de779-72cd-40b2-9fa7-e1da40599b2e/volumes"
Oct 30 03:47:37.706973 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.706943    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"]
Oct 30 03:47:37.707231 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707220    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo-gateway"
Oct 30 03:47:37.707276 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707233    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo-gateway"
Oct 30 03:47:37.707276 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707256    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerName="tempo"
Oct 30 03:47:37.707276 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707261    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerName="tempo"
Oct 30 03:47:37.707276 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707273    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707278    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707286    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo-gateway-opa"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707293    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo-gateway-opa"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707301    4334 cpu_manager.go:414] "containerMap: removing container" podUID="88e9b991-653f-49c2-8628-9e198a610f1e" containerName="otc-container"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707306    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e9b991-653f-49c2-8628-9e198a610f1e" containerName="otc-container"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707349    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo-gateway"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707359    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40f30ad-3443-4791-beaa-85eebdfcecba" containerName="tempo"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707366    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707373    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52de779-72cd-40b2-9fa7-e1da40599b2e" containerName="tempo-gateway-opa"
Oct 30 03:47:37.707405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.707379    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e9b991-653f-49c2-8628-9e198a610f1e" containerName="otc-container"
Oct 30 03:47:37.712065 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.712043    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.729562 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.729539    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"]
Oct 30 03:47:37.740925 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.740896    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:47:37.741048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.740933    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:47:37.741048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.741010    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:47:37.741191 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.741061    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\""
Oct 30 03:47:37.741289 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.741262    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:47:37.741403 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.741344    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-nljk2\""
Oct 30 03:47:37.741476 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.741404    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\""
Oct 30 03:47:37.758607 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.758587    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"]
Oct 30 03:47:37.761972 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.761957    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.781390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.781367    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"]
Oct 30 03:47:37.790954 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.790928    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"]
Oct 30 03:47:37.794340 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.794325    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:37.811414 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.811387    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"]
Oct 30 03:47:37.830585 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.830563    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 03:47:37.830698 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.830592    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 03:47:37.830698 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.830598    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 03:47:37.830698 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.830638    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 03:47:37.830884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.830742    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-rhftg\""
Oct 30 03:47:37.830884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.830792    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 03:47:37.830884 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.830853    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-r2sg4\""
Oct 30 03:47:37.848919 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.848897    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcm2\" (UniqueName: \"kubernetes.io/projected/b7c2ad8a-19bf-4d26-9b15-31b369395aed-kube-api-access-jdcm2\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.849031 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.848936    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.849031 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.848959    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.849156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.849046    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.849156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.849110    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b7c2ad8a-19bf-4d26-9b15-31b369395aed-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.849156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.849135    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.849282 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.849164    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.849282 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.849187    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzbh\" (UniqueName: \"kubernetes.io/projected/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-api-access-7bzbh\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.849282 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.849261    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.849374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.849282    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.949907 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.949872    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.949907 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.949911    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.950121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.949933    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcm2\" (UniqueName: \"kubernetes.io/projected/b7c2ad8a-19bf-4d26-9b15-31b369395aed-kube-api-access-jdcm2\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.950121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.949956    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfgp\" (UniqueName: \"kubernetes.io/projected/90ab7a71-1334-4148-a65f-f7a703c70558-kube-api-access-ckfgp\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:37.950121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950001    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.950121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950023    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.950121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950049    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.950394 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950127    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b7c2ad8a-19bf-4d26-9b15-31b369395aed-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.950394 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950179    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.950603 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950574    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.950682 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950632    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:37.950738 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950689    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.950802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950738    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.950802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950775    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzbh\" (UniqueName: \"kubernetes.io/projected/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-api-access-7bzbh\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.950917 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950802    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90ab7a71-1334-4148-a65f-f7a703c70558-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:37.950917 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.950859    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.951316 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.951290    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.952711 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.952686    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.952786 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.952717    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b7c2ad8a-19bf-4d26-9b15-31b369395aed-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.952786 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.952730    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.953023 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.953006    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:37.970302 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.970246    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcm2\" (UniqueName: \"kubernetes.io/projected/b7c2ad8a-19bf-4d26-9b15-31b369395aed-kube-api-access-jdcm2\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-cvdhz\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:37.971055 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:37.971034    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzbh\" (UniqueName: \"kubernetes.io/projected/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-api-access-7bzbh\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v5bfh\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:38.024763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.024730    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:38.051516 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.051488    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:38.051631 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.051544    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90ab7a71-1334-4148-a65f-f7a703c70558-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:38.051631 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.051605    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfgp\" (UniqueName: \"kubernetes.io/projected/90ab7a71-1334-4148-a65f-f7a703c70558-kube-api-access-ckfgp\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:38.051631 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:38.051624    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found
Oct 30 03:47:38.051798 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:38.051680    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls podName:90ab7a71-1334-4148-a65f-f7a703c70558 nodeName:}" failed. No retries permitted until 2026-10-30 03:47:38.551664424 +0000 UTC m=+3937.558254676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls") pod "data-science-thanos-querier-proxy-58d8d65bf9-sbk26" (UID: "90ab7a71-1334-4148-a65f-f7a703c70558") : secret "data-science-thanos-querier-proxy-tls" not found
Oct 30 03:47:38.052419 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.052396    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90ab7a71-1334-4148-a65f-f7a703c70558-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:38.076625 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.076598    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:47:38.077371 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.077348    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfgp\" (UniqueName: \"kubernetes.io/projected/90ab7a71-1334-4148-a65f-f7a703c70558-kube-api-access-ckfgp\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:38.321871 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.321833    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"]
Oct 30 03:47:38.323841 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:47:38.323815    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e7b6a6_ef32_4784_b89c_77eaaaa84d07.slice/crio-c565e45c73743bdec7199e25f056800289117a99d0cb30374688dd41cb31d8f9 WatchSource:0}: Error finding container c565e45c73743bdec7199e25f056800289117a99d0cb30374688dd41cb31d8f9: Status 404 returned error can't find the container with id c565e45c73743bdec7199e25f056800289117a99d0cb30374688dd41cb31d8f9
Oct 30 03:47:38.379289 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.379260    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"]
Oct 30 03:47:38.381731 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:47:38.381706    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c2ad8a_19bf_4d26_9b15_31b369395aed.slice/crio-cd7a4a508b72b847afc1169e4c496c4ac1078e502017eb2388cdd3aa9d1a3075 WatchSource:0}: Error finding container cd7a4a508b72b847afc1169e4c496c4ac1078e502017eb2388cdd3aa9d1a3075: Status 404 returned error can't find the container with id cd7a4a508b72b847afc1169e4c496c4ac1078e502017eb2388cdd3aa9d1a3075
Oct 30 03:47:38.557096 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.557072    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:38.559216 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.559192    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-sbk26\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:38.709429 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.709394    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:47:38.742694 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.742649    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" event={"ID":"b7c2ad8a-19bf-4d26-9b15-31b369395aed","Type":"ContainerStarted","Data":"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7"}
Oct 30 03:47:38.742867 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.742726    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" event={"ID":"b7c2ad8a-19bf-4d26-9b15-31b369395aed","Type":"ContainerStarted","Data":"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315"}
Oct 30 03:47:38.742867 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.742757    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" event={"ID":"b7c2ad8a-19bf-4d26-9b15-31b369395aed","Type":"ContainerStarted","Data":"cd7a4a508b72b847afc1169e4c496c4ac1078e502017eb2388cdd3aa9d1a3075"}
Oct 30 03:47:38.744197 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.744167    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh" event={"ID":"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07","Type":"ContainerStarted","Data":"c565e45c73743bdec7199e25f056800289117a99d0cb30374688dd41cb31d8f9"}
Oct 30 03:47:38.788498 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:38.788425    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" podStartSLOduration=1.788409497 podStartE2EDuration="1.788409497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:38.787582578 +0000 UTC m=+3937.794172833" watchObservedRunningTime="2026-10-30 03:47:38.788409497 +0000 UTC m=+3937.794999782"
Oct 30 03:47:39.007745 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.007721    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"]
Oct 30 03:47:39.009109 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:47:39.009079    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ab7a71_1334_4148_a65f_f7a703c70558.slice/crio-0ce0bd59e899072ac1df8479a5fd4f66d6540a2f5c522015e0ae7aea72638eda WatchSource:0}: Error finding container 0ce0bd59e899072ac1df8479a5fd4f66d6540a2f5c522015e0ae7aea72638eda: Status 404 returned error can't find the container with id 0ce0bd59e899072ac1df8479a5fd4f66d6540a2f5c522015e0ae7aea72638eda
Oct 30 03:47:39.298056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.297973    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:39.301839 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.301814    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.318783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.318586    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:47:39.318783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.318665    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:47:39.318783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.318665    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:47:39.318783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.318757    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:47:39.319040 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.318819    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:47:39.319663 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.319407    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:47:39.323116 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.323092    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:39.364150 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.364122    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.364293 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.364156    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.364293 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.364175    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.364293 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.364275    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c6lm\" (UniqueName: \"kubernetes.io/projected/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-kube-api-access-5c6lm\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.364429 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.364393    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.364719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.364479    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.465105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.465069    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.465259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.465115    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.465259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.465138    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.465259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.465181    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c6lm\" (UniqueName: \"kubernetes.io/projected/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-kube-api-access-5c6lm\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.465259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.465210    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.465259 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.465236    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.466061 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.466035    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.468427 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.468401    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.468650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.468615    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.468850 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.468829    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.469216 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.469197    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.485677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.485656    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c6lm\" (UniqueName: \"kubernetes.io/projected/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-kube-api-access-5c6lm\") pod \"data-science-collector-collector-0\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.615882 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.615819    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:47:39.752531 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.752490    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" event={"ID":"90ab7a71-1334-4148-a65f-f7a703c70558","Type":"ContainerStarted","Data":"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498"}
Oct 30 03:47:39.752689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.752575    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" event={"ID":"90ab7a71-1334-4148-a65f-f7a703c70558","Type":"ContainerStarted","Data":"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2"}
Oct 30 03:47:39.752689 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.752609    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" event={"ID":"90ab7a71-1334-4148-a65f-f7a703c70558","Type":"ContainerStarted","Data":"0ce0bd59e899072ac1df8479a5fd4f66d6540a2f5c522015e0ae7aea72638eda"}
Oct 30 03:47:39.753977 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.753939    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh" event={"ID":"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07","Type":"ContainerStarted","Data":"39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289"}
Oct 30 03:47:39.794433 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.794385    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" podStartSLOduration=2.79436923 podStartE2EDuration="2.79436923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:39.793390754 +0000 UTC m=+3938.799981009" watchObservedRunningTime="2026-10-30 03:47:39.79436923 +0000 UTC m=+3938.800959485"
Oct 30 03:47:39.836401 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.836349    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh" podStartSLOduration=2.836335853 podStartE2EDuration="2.836335853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:39.835883633 +0000 UTC m=+3938.842473889" watchObservedRunningTime="2026-10-30 03:47:39.836335853 +0000 UTC m=+3938.842926103"
Oct 30 03:47:39.917118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.917094    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:47:39.919107 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:47:39.919079    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb2dfb2_de29_4484_8e23_fbe548db7ff1.slice/crio-565e96b7491805696a301d9e1354126ec9974fbb20707ced413e6fa04e452b2b WatchSource:0}: Error finding container 565e96b7491805696a301d9e1354126ec9974fbb20707ced413e6fa04e452b2b: Status 404 returned error can't find the container with id 565e96b7491805696a301d9e1354126ec9974fbb20707ced413e6fa04e452b2b
Oct 30 03:47:39.924646 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.924609    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:47:39.924726 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:39.924675    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:47:40.761028 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:40.760989    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"bdb2dfb2-de29-4484-8e23-fbe548db7ff1","Type":"ContainerStarted","Data":"7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa"}
Oct 30 03:47:40.761556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:40.761048    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"bdb2dfb2-de29-4484-8e23-fbe548db7ff1","Type":"ContainerStarted","Data":"565e96b7491805696a301d9e1354126ec9974fbb20707ced413e6fa04e452b2b"}
Oct 30 03:47:40.761556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:40.761391    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:40.806290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:40.806245    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.8062335809999999 podStartE2EDuration="1.806233581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:40.805493661 +0000 UTC m=+3939.812083918" watchObservedRunningTime="2026-10-30 03:47:40.806233581 +0000 UTC m=+3939.812823834"
Oct 30 03:47:42.875054 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.875016    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:47:42.878783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.878760    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:42.913889 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.913860    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:47:42.917676 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.917654    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:47:42.917794 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.917752    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.930775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.930753    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:47:42.989861 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.989828    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:47:42.990089 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990072    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:47:42.990089 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990072    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:47:42.990211 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990091    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:47:42.990267 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990237    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:47:42.990314 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990264    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 03:47:42.990314 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990270    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 03:47:42.990403 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990341    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:47:42.990488 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990441    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:47:42.990548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990486    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-fp6sj\""
Oct 30 03:47:42.990548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990508    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-5cs6w\""
Oct 30 03:47:42.990548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990536    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 03:47:42.990657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990564    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:47:42.990657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.990589    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:47:42.996055 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996035    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvpn\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-kube-api-access-ncvpn\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:42.996160 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996074    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996160 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996094    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:42.996265 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996207    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996310 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996296    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996354 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996328    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996354 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996350    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:42.996469 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996367    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:42.996469 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996409    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:42.996469 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996441    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996558    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:42.996649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996591    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996620    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:42.996783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996654    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996682    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996735    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996765    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996942 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996791    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:42.996942 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:42.996825    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8xb\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-kube-api-access-qb8xb\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.097892 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.097862    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.098024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.097895    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.098024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.097921    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.098024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.097945    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.097975    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.098255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098081    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098126    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.098255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098154    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098182    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098212    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098239    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098602 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098270    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098602 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098294    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8xb\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-kube-api-access-qb8xb\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098602 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098345    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvpn\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-kube-api-access-ncvpn\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.098789 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098765    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098844 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098809    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.098902 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098882    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.098969 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098927    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.099025 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.098970    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.099140 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.099117    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.101995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.101249    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.101995 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.101575    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.103667 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.102694    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.103667 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.103490    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.104765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.103825    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.104765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.104373    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.104765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.104401    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.104946 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.104918    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.105323 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.105297    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.105428 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.105393    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.105532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.105506    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.105797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.105770    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.105797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.105786    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.106156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.106132    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.106935 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.106911    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.107396 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.107379    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.122017 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.121996    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvpn\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-kube-api-access-ncvpn\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.122187 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.122171    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8xb\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-kube-api-access-qb8xb\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:43.195911 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.195891    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:43.512366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.512343    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:47:43.513784 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:47:43.513757    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6849b93c_0a73_448c_9021_d548ecfd2ba1.slice/crio-5f14e011d97934cd6911d720a0c515e5a7112962e8a22b257eeb7412745961bc WatchSource:0}: Error finding container 5f14e011d97934cd6911d720a0c515e5a7112962e8a22b257eeb7412745961bc: Status 404 returned error can't find the container with id 5f14e011d97934cd6911d720a0c515e5a7112962e8a22b257eeb7412745961bc
Oct 30 03:47:43.776287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:43.776200    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6849b93c-0a73-448c-9021-d548ecfd2ba1","Type":"ContainerStarted","Data":"5f14e011d97934cd6911d720a0c515e5a7112962e8a22b257eeb7412745961bc"}
Oct 30 03:47:44.782111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:44.782075    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6849b93c-0a73-448c-9021-d548ecfd2ba1","Type":"ContainerStarted","Data":"e50474ecc1998acb08bb4d92d132741c508ffff4c7b013a42d456da64902dec9"}
Oct 30 03:47:46.767939 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:46.767909    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:47:47.798260 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:47.798231    4334 generic.go:356] "Generic (PLEG): container finished" podID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerID="e50474ecc1998acb08bb4d92d132741c508ffff4c7b013a42d456da64902dec9" exitCode=0
Oct 30 03:47:47.798854 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:47.798304    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6849b93c-0a73-448c-9021-d548ecfd2ba1","Type":"ContainerDied","Data":"e50474ecc1998acb08bb4d92d132741c508ffff4c7b013a42d456da64902dec9"}
Oct 30 03:47:48.805735 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:48.805701    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6849b93c-0a73-448c-9021-d548ecfd2ba1","Type":"ContainerStarted","Data":"81dc8ae1ccc783a6f51a083458e381ad26dd9acc2538a6ab13cdb4ddf105ef37"}
Oct 30 03:47:48.806156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:48.805761    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6849b93c-0a73-448c-9021-d548ecfd2ba1","Type":"ContainerStarted","Data":"1eb61eb42097b219ca45152a04c7822723f338df4b74a0d2c221e428b1cb64cb"}
Oct 30 03:47:48.806156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:48.805955    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:48.876848 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:48.876798    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.49091493 podStartE2EDuration="6.876784673s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385869743s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:48.863013786 +0000 UTC m=+3947.869604041" watchObservedRunningTime="2026-10-30 03:47:48.876784673 +0000 UTC m=+3947.883374927"
Oct 30 03:47:50.979105 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:47:50.979073    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:47:52.480836 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:52.480802    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:52.511010 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:52.510978    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:52.581513 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:52.581483    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:52.581693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:52.581607    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:52.596542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:52.596518    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:52.698994 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:52.698959    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:52.725599 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:52.725576    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:52.838141 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:52.838065    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:47:53.165472 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:53.165431    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:47:53.182070 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:47:53.182043    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde58ed3d_7cb5_4588_9f3e_69f0cbc2be18.slice/crio-e205f6f1b98a827324ff749f8af88cf387b3d92b0e0c33fd1d9899fa2af77c85 WatchSource:0}: Error finding container e205f6f1b98a827324ff749f8af88cf387b3d92b0e0c33fd1d9899fa2af77c85: Status 404 returned error can't find the container with id e205f6f1b98a827324ff749f8af88cf387b3d92b0e0c33fd1d9899fa2af77c85
Oct 30 03:47:53.831149 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:53.831064    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerStarted","Data":"c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39"}
Oct 30 03:47:53.831653 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:53.831176    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerStarted","Data":"e205f6f1b98a827324ff749f8af88cf387b3d92b0e0c33fd1d9899fa2af77c85"}
Oct 30 03:47:54.811712 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:54.811678    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:47:57.852549 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:57.852514    4334 generic.go:356] "Generic (PLEG): container finished" podID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerID="c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39" exitCode=0
Oct 30 03:47:57.852988 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:57.852588    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerDied","Data":"c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39"}
Oct 30 03:47:58.860532 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:58.860500    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerStarted","Data":"017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c"}
Oct 30 03:47:58.860940 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:58.860568    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerStarted","Data":"3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce"}
Oct 30 03:47:58.860940 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:58.860592    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerStarted","Data":"5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db"}
Oct 30 03:47:58.938360 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:47:58.938314    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=13.653069666 podStartE2EDuration="16.938302551s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285232885s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:58.924563664 +0000 UTC m=+3957.931153917" watchObservedRunningTime="2026-10-30 03:47:58.938302551 +0000 UTC m=+3957.944892842"
Oct 30 03:48:02.838710 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:02.838636    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:48:07.838405 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:07.838365    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:48:07.843655 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:07.843633    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:48:07.908543 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:07.908522    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:48:18.488642 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:18.488596    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:18.489146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:18.488876    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="bdb2dfb2-de29-4484-8e23-fbe548db7ff1" containerName="otc-container" containerID="cri-o://7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa" gracePeriod=30
Oct 30 03:48:20.737384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.737360    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:20.823490 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.823463    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-monitoring-tls-certs\") pod \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") "
Oct 30 03:48:20.823644 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.823520    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otlp-tls-certs\") pod \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") "
Oct 30 03:48:20.823644 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.823559    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-kube-api-access-5c6lm\" (UniqueName: \"kubernetes.io/projected/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-kube-api-access-5c6lm\") pod \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") "
Oct 30 03:48:20.823644 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.823620    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-data-science-collector-ta-client-cert\") pod \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") "
Oct 30 03:48:20.823777 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.823649    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-tls-certs\") pod \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") "
Oct 30 03:48:20.823777 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.823754    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otc-internal\") pod \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\" (UID: \"bdb2dfb2-de29-4484-8e23-fbe548db7ff1\") "
Oct 30 03:48:20.824216 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.824182    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otc-internal" pod "bdb2dfb2-de29-4484-8e23-fbe548db7ff1" (UID: "bdb2dfb2-de29-4484-8e23-fbe548db7ff1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:20.825856 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.825823    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otlp-tls-certs" pod "bdb2dfb2-de29-4484-8e23-fbe548db7ff1" (UID: "bdb2dfb2-de29-4484-8e23-fbe548db7ff1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:20.825983 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.825888    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-data-science-collector-ta-client-cert" pod "bdb2dfb2-de29-4484-8e23-fbe548db7ff1" (UID: "bdb2dfb2-de29-4484-8e23-fbe548db7ff1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:20.825983 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.825915    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-kube-api-access-5c6lm" pod "bdb2dfb2-de29-4484-8e23-fbe548db7ff1" (UID: "bdb2dfb2-de29-4484-8e23-fbe548db7ff1"). InnerVolumeSpecName "kube-api-access-5c6lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:20.826201 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.826178    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-tls-certs" pod "bdb2dfb2-de29-4484-8e23-fbe548db7ff1" (UID: "bdb2dfb2-de29-4484-8e23-fbe548db7ff1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:20.826247 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.826191    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-monitoring-tls-certs" pod "bdb2dfb2-de29-4484-8e23-fbe548db7ff1" (UID: "bdb2dfb2-de29-4484-8e23-fbe548db7ff1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:20.924865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.924843    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.924865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.924864    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c6lm\" (UniqueName: \"kubernetes.io/projected/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-kube-api-access-5c6lm\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.925021 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.924875    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.925021 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.924884    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.925021 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.924893    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.925021 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.924901    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb2dfb2-de29-4484-8e23-fbe548db7ff1-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.966517 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.966488    4334 generic.go:356] "Generic (PLEG): container finished" podID="bdb2dfb2-de29-4484-8e23-fbe548db7ff1" containerID="7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa" exitCode=0
Oct 30 03:48:20.966644 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.966560    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:20.966644 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.966566    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"bdb2dfb2-de29-4484-8e23-fbe548db7ff1","Type":"ContainerDied","Data":"7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa"}
Oct 30 03:48:20.966644 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.966622    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"bdb2dfb2-de29-4484-8e23-fbe548db7ff1","Type":"ContainerDied","Data":"565e96b7491805696a301d9e1354126ec9974fbb20707ced413e6fa04e452b2b"}
Oct 30 03:48:20.966763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.966646    4334 scope.go:122] "RemoveContainer" containerID="7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa"
Oct 30 03:48:20.975582 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.975520    4334 scope.go:122] "RemoveContainer" containerID="7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa"
Oct 30 03:48:20.975805 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:48:20.975778    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa\": container with ID starting with 7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa not found: ID does not exist" containerID="7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa"
Oct 30 03:48:20.975848 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:20.975819    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa"} err="failed to get container status \"7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa\": rpc error: code = NotFound desc = could not find container \"7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa\": container with ID starting with 7a3281809894a769883b393d36724ffbf66837545edc87fab64860edd3402faa not found: ID does not exist"
Oct 30 03:48:20.983332 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:48:20.983262    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:48:21.019546 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.019521    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:21.024384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.024362    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:21.060760 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.060740    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:21.061164 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.061144    4334 cpu_manager.go:414] "containerMap: removing container" podUID="bdb2dfb2-de29-4484-8e23-fbe548db7ff1" containerName="otc-container"
Oct 30 03:48:21.061164 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.061163    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb2dfb2-de29-4484-8e23-fbe548db7ff1" containerName="otc-container"
Oct 30 03:48:21.061334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.061247    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb2dfb2-de29-4484-8e23-fbe548db7ff1" containerName="otc-container"
Oct 30 03:48:21.064078 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.064065    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.082993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.082963    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:48:21.085570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.085388    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:48:21.085570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.085498    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:48:21.085741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.085683    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:48:21.085937 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.085792    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:48:21.085937 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.085807    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:21.085937 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.085913    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:48:21.158626 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.158597    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb2dfb2-de29-4484-8e23-fbe548db7ff1" path="/var/lib/kubelet/pods/bdb2dfb2-de29-4484-8e23-fbe548db7ff1/volumes"
Oct 30 03:48:21.228023 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.227953    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f0d8529-be47-4114-b2db-07c3675c154d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.228170 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.228027    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.228239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.228169    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.228239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.228219    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hhv\" (UniqueName: \"kubernetes.io/projected/3f0d8529-be47-4114-b2db-07c3675c154d-kube-api-access-z7hhv\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.228341 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.228275    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.228417 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.228399    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.329786 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.329749    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.329951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.329795    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.329951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.329825    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f0d8529-be47-4114-b2db-07c3675c154d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.329951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.329865    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.329951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.329931    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.330179 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.329970    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hhv\" (UniqueName: \"kubernetes.io/projected/3f0d8529-be47-4114-b2db-07c3675c154d-kube-api-access-z7hhv\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.330765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.330733    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f0d8529-be47-4114-b2db-07c3675c154d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.332487 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.332463    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.332626 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.332604    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.332859 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.332832    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.332978 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.332832    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.352709 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.352686    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hhv\" (UniqueName: \"kubernetes.io/projected/3f0d8529-be47-4114-b2db-07c3675c154d-kube-api-access-z7hhv\") pod \"data-science-collector-collector-0\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.381098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.381077    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:21.685392 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.685361    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:21.689050 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:48:21.689025    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0d8529_be47_4114_b2db_07c3675c154d.slice/crio-0db897031c0bf82d3a6bb87a25fe667bd8b65fa58b6ac367fa8bd1e9325f7169 WatchSource:0}: Error finding container 0db897031c0bf82d3a6bb87a25fe667bd8b65fa58b6ac367fa8bd1e9325f7169: Status 404 returned error can't find the container with id 0db897031c0bf82d3a6bb87a25fe667bd8b65fa58b6ac367fa8bd1e9325f7169
Oct 30 03:48:21.691773 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.691739    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:48:21.691880 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.691810    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:48:21.974002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.973910    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3f0d8529-be47-4114-b2db-07c3675c154d","Type":"ContainerStarted","Data":"54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b"}
Oct 30 03:48:21.974002 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:21.973966    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3f0d8529-be47-4114-b2db-07c3675c154d","Type":"ContainerStarted","Data":"0db897031c0bf82d3a6bb87a25fe667bd8b65fa58b6ac367fa8bd1e9325f7169"}
Oct 30 03:48:22.016682 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:22.016626    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.016613179 podStartE2EDuration="1.016613179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:22.014275716 +0000 UTC m=+3981.020865971" watchObservedRunningTime="2026-10-30 03:48:22.016613179 +0000 UTC m=+3981.023203432"
Oct 30 03:48:42.609784 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:42.609741    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:42.610323 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:42.610018    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="3f0d8529-be47-4114-b2db-07c3675c154d" containerName="otc-container" containerID="cri-o://54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b" gracePeriod=30
Oct 30 03:48:43.855410 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.855376    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:43.903400 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.903374    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0d8529-be47-4114-b2db-07c3675c154d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f0d8529-be47-4114-b2db-07c3675c154d-otc-internal\") pod \"3f0d8529-be47-4114-b2db-07c3675c154d\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") "
Oct 30 03:48:43.903562 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.903416    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-data-science-collector-ta-client-cert\") pod \"3f0d8529-be47-4114-b2db-07c3675c154d\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") "
Oct 30 03:48:43.903562 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.903486    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0d8529-be47-4114-b2db-07c3675c154d-kube-api-access-z7hhv\" (UniqueName: \"kubernetes.io/projected/3f0d8529-be47-4114-b2db-07c3675c154d-kube-api-access-z7hhv\") pod \"3f0d8529-be47-4114-b2db-07c3675c154d\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") "
Oct 30 03:48:43.903562 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.903513    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-otlp-tls-certs\") pod \"3f0d8529-be47-4114-b2db-07c3675c154d\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") "
Oct 30 03:48:43.903562 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.903551    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-tls-certs\") pod \"3f0d8529-be47-4114-b2db-07c3675c154d\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") "
Oct 30 03:48:43.903804 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.903589    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-monitoring-tls-certs\") pod \"3f0d8529-be47-4114-b2db-07c3675c154d\" (UID: \"3f0d8529-be47-4114-b2db-07c3675c154d\") "
Oct 30 03:48:43.903865 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.903835    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0d8529-be47-4114-b2db-07c3675c154d-otc-internal" pod "3f0d8529-be47-4114-b2db-07c3675c154d" (UID: "3f0d8529-be47-4114-b2db-07c3675c154d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:43.906019 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.905992    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-data-science-collector-ta-client-cert" pod "3f0d8529-be47-4114-b2db-07c3675c154d" (UID: "3f0d8529-be47-4114-b2db-07c3675c154d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:43.906719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.906694    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-tls-certs" pod "3f0d8529-be47-4114-b2db-07c3675c154d" (UID: "3f0d8529-be47-4114-b2db-07c3675c154d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:43.907314 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.907276    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-monitoring-tls-certs" pod "3f0d8529-be47-4114-b2db-07c3675c154d" (UID: "3f0d8529-be47-4114-b2db-07c3675c154d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:43.907419 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.907395    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0d8529-be47-4114-b2db-07c3675c154d-kube-api-access-z7hhv" pod "3f0d8529-be47-4114-b2db-07c3675c154d" (UID: "3f0d8529-be47-4114-b2db-07c3675c154d"). InnerVolumeSpecName "kube-api-access-z7hhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:43.907521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:43.907395    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-otlp-tls-certs" pod "3f0d8529-be47-4114-b2db-07c3675c154d" (UID: "3f0d8529-be47-4114-b2db-07c3675c154d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:44.004834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.004773    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f0d8529-be47-4114-b2db-07c3675c154d-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:44.004834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.004799    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:44.004834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.004810    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hhv\" (UniqueName: \"kubernetes.io/projected/3f0d8529-be47-4114-b2db-07c3675c154d-kube-api-access-z7hhv\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:44.004834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.004821    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:44.004834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.004831    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:44.004834 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.004839    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f0d8529-be47-4114-b2db-07c3675c154d-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:44.082236 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.082209    4334 generic.go:356] "Generic (PLEG): container finished" podID="3f0d8529-be47-4114-b2db-07c3675c154d" containerID="54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b" exitCode=0
Oct 30 03:48:44.082360 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.082254    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.082360 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.082303    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3f0d8529-be47-4114-b2db-07c3675c154d","Type":"ContainerDied","Data":"54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b"}
Oct 30 03:48:44.082360 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.082356    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3f0d8529-be47-4114-b2db-07c3675c154d","Type":"ContainerDied","Data":"0db897031c0bf82d3a6bb87a25fe667bd8b65fa58b6ac367fa8bd1e9325f7169"}
Oct 30 03:48:44.082556 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.082378    4334 scope.go:122] "RemoveContainer" containerID="54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b"
Oct 30 03:48:44.092027 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.092010    4334 scope.go:122] "RemoveContainer" containerID="54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b"
Oct 30 03:48:44.092307 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:48:44.092286    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b\": container with ID starting with 54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b not found: ID does not exist" containerID="54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b"
Oct 30 03:48:44.092352 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.092316    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b"} err="failed to get container status \"54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b\": rpc error: code = NotFound desc = could not find container \"54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b\": container with ID starting with 54121348f7bc0dccd3e5b94a85c4a573b1e38e37ce7391b1c20198f78a3af23b not found: ID does not exist"
Oct 30 03:48:44.140366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.140335    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:44.147992 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.147971    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:44.190510 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.190484    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:44.190920 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.190902    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3f0d8529-be47-4114-b2db-07c3675c154d" containerName="otc-container"
Oct 30 03:48:44.191006 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.190922    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0d8529-be47-4114-b2db-07c3675c154d" containerName="otc-container"
Oct 30 03:48:44.191063 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.191016    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0d8529-be47-4114-b2db-07c3675c154d" containerName="otc-container"
Oct 30 03:48:44.198862 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.198843    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.205595 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.205569    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.205678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.205614    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.205717 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.205680    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.205757 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.205734    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.205878 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.205856    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncb2c\" (UniqueName: \"kubernetes.io/projected/a2f015f0-2dbf-45fd-8844-27ad0edc2596-kube-api-access-ncb2c\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.205951 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.205924    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.211800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.211778    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:44.222366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.222345    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\""
Oct 30 03:48:44.222511 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.222418    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:48:44.222606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.222500    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:48:44.222606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.222431    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:48:44.222606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.222538    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:48:44.222729 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.222709    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:48:44.307042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.306973    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.307042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.307006    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.307042 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.307032    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.307290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.307091    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.307290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.307139    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncb2c\" (UniqueName: \"kubernetes.io/projected/a2f015f0-2dbf-45fd-8844-27ad0edc2596-kube-api-access-ncb2c\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.307290 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.307258    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.307808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.307785    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.309622 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.309596    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.309824 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.309803    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.309890 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.309818    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.310024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.310009    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.328130 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.328111    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncb2c\" (UniqueName: \"kubernetes.io/projected/a2f015f0-2dbf-45fd-8844-27ad0edc2596-kube-api-access-ncb2c\") pod \"data-science-collector-collector-0\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.511367 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.511339    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:44.809130 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.809095    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:44.812844 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:48:44.812788    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f015f0_2dbf_45fd_8844_27ad0edc2596.slice/crio-7b222f8002c704861a9c3267eef9fe520eb65a8a8920599fb2627346f9178d10 WatchSource:0}: Error finding container 7b222f8002c704861a9c3267eef9fe520eb65a8a8920599fb2627346f9178d10: Status 404 returned error can't find the container with id 7b222f8002c704861a9c3267eef9fe520eb65a8a8920599fb2627346f9178d10
Oct 30 03:48:44.815324 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.815291    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:48:44.815387 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:44.815362    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:48:45.091122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:45.091031    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a2f015f0-2dbf-45fd-8844-27ad0edc2596","Type":"ContainerStarted","Data":"1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca"}
Oct 30 03:48:45.091122 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:45.091115    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a2f015f0-2dbf-45fd-8844-27ad0edc2596","Type":"ContainerStarted","Data":"7b222f8002c704861a9c3267eef9fe520eb65a8a8920599fb2627346f9178d10"}
Oct 30 03:48:45.135211 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:45.135153    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.135137395 podStartE2EDuration="1.135137395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:45.133756101 +0000 UTC m=+4004.140346355" watchObservedRunningTime="2026-10-30 03:48:45.135137395 +0000 UTC m=+4004.141727649"
Oct 30 03:48:45.160733 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:45.160701    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0d8529-be47-4114-b2db-07c3675c154d" path="/var/lib/kubelet/pods/3f0d8529-be47-4114-b2db-07c3675c154d/volumes"
Oct 30 03:48:50.978551 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:48:50.978510    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:48:52.132232 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:52.132188    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:52.132779 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:52.132420    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="a2f015f0-2dbf-45fd-8844-27ad0edc2596" containerName="otc-container" containerID="cri-o://1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca" gracePeriod=30
Oct 30 03:48:53.378827 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.378801    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:53.486598 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.486525    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-monitoring-tls-certs\") pod \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") "
Oct 30 03:48:53.486752 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.486630    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-data-science-collector-ta-client-cert\") pod \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") "
Oct 30 03:48:53.486752 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.486655    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f015f0-2dbf-45fd-8844-27ad0edc2596-kube-api-access-ncb2c\" (UniqueName: \"kubernetes.io/projected/a2f015f0-2dbf-45fd-8844-27ad0edc2596-kube-api-access-ncb2c\") pod \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") "
Oct 30 03:48:53.486752 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.486674    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otlp-tls-certs\") pod \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") "
Oct 30 03:48:53.486752 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.486695    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otc-internal\") pod \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") "
Oct 30 03:48:53.486752 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.486710    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-tls-certs\") pod \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\" (UID: \"a2f015f0-2dbf-45fd-8844-27ad0edc2596\") "
Oct 30 03:48:53.487129 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.487085    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otc-internal" pod "a2f015f0-2dbf-45fd-8844-27ad0edc2596" (UID: "a2f015f0-2dbf-45fd-8844-27ad0edc2596"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:53.488835 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.488806    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-monitoring-tls-certs" pod "a2f015f0-2dbf-45fd-8844-27ad0edc2596" (UID: "a2f015f0-2dbf-45fd-8844-27ad0edc2596"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:53.488968 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.488845    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-data-science-collector-ta-client-cert" pod "a2f015f0-2dbf-45fd-8844-27ad0edc2596" (UID: "a2f015f0-2dbf-45fd-8844-27ad0edc2596"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:53.488968 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.488915    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-tls-certs" pod "a2f015f0-2dbf-45fd-8844-27ad0edc2596" (UID: "a2f015f0-2dbf-45fd-8844-27ad0edc2596"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:53.489231 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.489215    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otlp-tls-certs" pod "a2f015f0-2dbf-45fd-8844-27ad0edc2596" (UID: "a2f015f0-2dbf-45fd-8844-27ad0edc2596"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:53.489296 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.489219    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f015f0-2dbf-45fd-8844-27ad0edc2596-kube-api-access-ncb2c" pod "a2f015f0-2dbf-45fd-8844-27ad0edc2596" (UID: "a2f015f0-2dbf-45fd-8844-27ad0edc2596"). InnerVolumeSpecName "kube-api-access-ncb2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:53.587315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.587292    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:53.587315 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.587316    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncb2c\" (UniqueName: \"kubernetes.io/projected/a2f015f0-2dbf-45fd-8844-27ad0edc2596-kube-api-access-ncb2c\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:53.587500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.587327    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:53.587500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.587337    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2f015f0-2dbf-45fd-8844-27ad0edc2596-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:53.587500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.587349    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:53.587500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:53.587359    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f015f0-2dbf-45fd-8844-27ad0edc2596-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:48:54.145634 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.145596    4334 generic.go:356] "Generic (PLEG): container finished" podID="a2f015f0-2dbf-45fd-8844-27ad0edc2596" containerID="1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca" exitCode=0
Oct 30 03:48:54.145826 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.145665    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:48:54.145826 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.145673    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a2f015f0-2dbf-45fd-8844-27ad0edc2596","Type":"ContainerDied","Data":"1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca"}
Oct 30 03:48:54.145826 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.145821    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a2f015f0-2dbf-45fd-8844-27ad0edc2596","Type":"ContainerDied","Data":"7b222f8002c704861a9c3267eef9fe520eb65a8a8920599fb2627346f9178d10"}
Oct 30 03:48:54.145996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.145855    4334 scope.go:122] "RemoveContainer" containerID="1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca"
Oct 30 03:48:54.155377 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.155363    4334 scope.go:122] "RemoveContainer" containerID="1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca"
Oct 30 03:48:54.155672 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:48:54.155653    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca\": container with ID starting with 1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca not found: ID does not exist" containerID="1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca"
Oct 30 03:48:54.155733 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.155679    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca"} err="failed to get container status \"1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca\": rpc error: code = NotFound desc = could not find container \"1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca\": container with ID starting with 1789532b3c96fcc6f637752d04b9b98e9c59658a297e8c9e61307f65ec13a5ca not found: ID does not exist"
Oct 30 03:48:54.208845 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.208823    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:54.222082 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:54.222061    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:48:55.158432 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:55.158400    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f015f0-2dbf-45fd-8844-27ad0edc2596" path="/var/lib/kubelet/pods/a2f015f0-2dbf-45fd-8844-27ad0edc2596/volumes"
Oct 30 03:48:59.220118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.220082    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:59.220649 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.220620    4334 cpu_manager.go:414] "containerMap: removing container" podUID="a2f015f0-2dbf-45fd-8844-27ad0edc2596" containerName="otc-container"
Oct 30 03:48:59.220709 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.220651    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f015f0-2dbf-45fd-8844-27ad0edc2596" containerName="otc-container"
Oct 30 03:48:59.220782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.220771    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f015f0-2dbf-45fd-8844-27ad0edc2596" containerName="otc-container"
Oct 30 03:48:59.225728 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.225708    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.242708 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.242676    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:59.253635 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.253617    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\""
Oct 30 03:48:59.255628 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.255605    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:48:59.255724 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.255627    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:48:59.255724 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.255691    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:48:59.266740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.266724    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:48:59.266840 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.266826    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:48:59.334970 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.334946    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.335104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.334980    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.335104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.334999    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkg69\" (UniqueName: \"kubernetes.io/projected/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-kube-api-access-fkg69\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.335104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.335019    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.335237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.335149    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.335237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.335179    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.436381 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.436356    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.436537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.436405    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.436537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.436434    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.436537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.436478    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkg69\" (UniqueName: \"kubernetes.io/projected/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-kube-api-access-fkg69\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.436537 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.436512    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.436783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.436567    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.437244 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.437220    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.439114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.439086    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.439238 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.439167    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.439238 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.439193    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.439414 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.439360    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.456943 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.456920    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkg69\" (UniqueName: \"kubernetes.io/projected/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-kube-api-access-fkg69\") pod \"data-science-collector-collector-1\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.539109 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.539041    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:48:59.836475 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.836385    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:48:59.841608 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:48:59.841571    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2828292_0d7e_420d_a5da_ff13c0bdcfa9.slice/crio-1f07b69d9e4ad197c710a540399d8ca618ea778dc63cca3d33ccb3beb9c1975c WatchSource:0}: Error finding container 1f07b69d9e4ad197c710a540399d8ca618ea778dc63cca3d33ccb3beb9c1975c: Status 404 returned error can't find the container with id 1f07b69d9e4ad197c710a540399d8ca618ea778dc63cca3d33ccb3beb9c1975c
Oct 30 03:48:59.843773 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.843731    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:48:59.843868 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:48:59.843812    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:49:00.176756 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:00.176712    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f2828292-0d7e-420d-a5da-ff13c0bdcfa9","Type":"ContainerStarted","Data":"074f1f661e51167d992832c14f6512d32548c4a08ed61676de7018c4823c2ea5"}
Oct 30 03:49:00.176920 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:00.176777    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f2828292-0d7e-420d-a5da-ff13c0bdcfa9","Type":"ContainerStarted","Data":"1f07b69d9e4ad197c710a540399d8ca618ea778dc63cca3d33ccb3beb9c1975c"}
Oct 30 03:49:00.219909 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:00.219859    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.219841686 podStartE2EDuration="1.219841686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:00.21820135 +0000 UTC m=+4019.224791604" watchObservedRunningTime="2026-10-30 03:49:00.219841686 +0000 UTC m=+4019.226431940"
Oct 30 03:49:07.092207 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.092178    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:07.092642 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.092375    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f2828292-0d7e-420d-a5da-ff13c0bdcfa9" containerName="otc-container" containerID="cri-o://074f1f661e51167d992832c14f6512d32548c4a08ed61676de7018c4823c2ea5" gracePeriod=30
Oct 30 03:49:07.218987 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.218951    4334 generic.go:356] "Generic (PLEG): container finished" podID="f2828292-0d7e-420d-a5da-ff13c0bdcfa9" containerID="074f1f661e51167d992832c14f6512d32548c4a08ed61676de7018c4823c2ea5" exitCode=0
Oct 30 03:49:07.219109 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.219011    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f2828292-0d7e-420d-a5da-ff13c0bdcfa9","Type":"ContainerDied","Data":"074f1f661e51167d992832c14f6512d32548c4a08ed61676de7018c4823c2ea5"}
Oct 30 03:49:07.342653 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.342597    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:07.405006 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.404974    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-kube-api-access-fkg69\" (UniqueName: \"kubernetes.io/projected/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-kube-api-access-fkg69\") pod \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") "
Oct 30 03:49:07.405178 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.405018    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otlp-tls-certs\") pod \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") "
Oct 30 03:49:07.405178 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.405140    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otc-internal\") pod \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") "
Oct 30 03:49:07.405296 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.405190    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-monitoring-tls-certs\") pod \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") "
Oct 30 03:49:07.405296 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.405231    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-data-science-collector-ta-client-cert\") pod \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") "
Oct 30 03:49:07.405296 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.405263    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-tls-certs\") pod \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\" (UID: \"f2828292-0d7e-420d-a5da-ff13c0bdcfa9\") "
Oct 30 03:49:07.405523 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.405500    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otc-internal" pod "f2828292-0d7e-420d-a5da-ff13c0bdcfa9" (UID: "f2828292-0d7e-420d-a5da-ff13c0bdcfa9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:07.405603 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.405584    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:07.407458 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.407394    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otlp-tls-certs" pod "f2828292-0d7e-420d-a5da-ff13c0bdcfa9" (UID: "f2828292-0d7e-420d-a5da-ff13c0bdcfa9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:07.407840 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.407814    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-monitoring-tls-certs" pod "f2828292-0d7e-420d-a5da-ff13c0bdcfa9" (UID: "f2828292-0d7e-420d-a5da-ff13c0bdcfa9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:07.408093 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.408070    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-kube-api-access-fkg69" pod "f2828292-0d7e-420d-a5da-ff13c0bdcfa9" (UID: "f2828292-0d7e-420d-a5da-ff13c0bdcfa9"). InnerVolumeSpecName "kube-api-access-fkg69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:07.408093 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.408071    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-data-science-collector-ta-client-cert" pod "f2828292-0d7e-420d-a5da-ff13c0bdcfa9" (UID: "f2828292-0d7e-420d-a5da-ff13c0bdcfa9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:07.408213 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.408088    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-tls-certs" pod "f2828292-0d7e-420d-a5da-ff13c0bdcfa9" (UID: "f2828292-0d7e-420d-a5da-ff13c0bdcfa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:07.506853 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.506819    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkg69\" (UniqueName: \"kubernetes.io/projected/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-kube-api-access-fkg69\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:07.506853 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.506850    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:07.507030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.506863    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:07.507030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.506878    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:07.507030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:07.506891    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2828292-0d7e-420d-a5da-ff13c0bdcfa9-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:08.225998 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.225967    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.225998 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.225980    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f2828292-0d7e-420d-a5da-ff13c0bdcfa9","Type":"ContainerDied","Data":"1f07b69d9e4ad197c710a540399d8ca618ea778dc63cca3d33ccb3beb9c1975c"}
Oct 30 03:49:08.226570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.226039    4334 scope.go:122] "RemoveContainer" containerID="074f1f661e51167d992832c14f6512d32548c4a08ed61676de7018c4823c2ea5"
Oct 30 03:49:08.279726 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.279692    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:08.293318 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.293290    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:08.324856 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.324830    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:08.325223 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.325209    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f2828292-0d7e-420d-a5da-ff13c0bdcfa9" containerName="otc-container"
Oct 30 03:49:08.325271 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.325225    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2828292-0d7e-420d-a5da-ff13c0bdcfa9" containerName="otc-container"
Oct 30 03:49:08.325326 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.325316    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2828292-0d7e-420d-a5da-ff13c0bdcfa9" containerName="otc-container"
Oct 30 03:49:08.329705 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.329687    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.345179 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.345150    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:08.345901 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.345881    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:49:08.345901 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.345884    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:49:08.346035 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.345947    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wtzv9\""
Oct 30 03:49:08.346035 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.345985    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:49:08.346111 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.346057    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:49:08.387264 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.387232    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:49:08.412765 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.412736    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.412913 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.412775    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fbbb\" (UniqueName: \"kubernetes.io/projected/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-kube-api-access-6fbbb\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.412913 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.412880    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.413034 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.412932    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.413034 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.413023    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.413147 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.413057    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.513938 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.513860    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.513938 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.513903    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.513938 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.513924    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.514189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.514039    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.514189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.514090    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.514189 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.514138    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fbbb\" (UniqueName: \"kubernetes.io/projected/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-kube-api-access-6fbbb\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.514691 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.514654    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.516738 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.516706    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.516879 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.516763    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.516879 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.516819    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.516879 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.516828    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.533606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.533583    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fbbb\" (UniqueName: \"kubernetes.io/projected/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-kube-api-access-6fbbb\") pod \"data-science-collector-collector-1\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.641853 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.641823    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:08.939307 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.939278    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:08.940899 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:49:08.940875    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5ceef3_a9de_4329_ac29_7d7430c5f03d.slice/crio-baa12ec055035bd380ffd65bf26f49951ebe67ff772ad02bf0273693587ecf9c WatchSource:0}: Error finding container baa12ec055035bd380ffd65bf26f49951ebe67ff772ad02bf0273693587ecf9c: Status 404 returned error can't find the container with id baa12ec055035bd380ffd65bf26f49951ebe67ff772ad02bf0273693587ecf9c
Oct 30 03:49:08.943161 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.943131    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:49:08.943257 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:08.943196    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:49:09.158943 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:09.158911    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2828292-0d7e-420d-a5da-ff13c0bdcfa9" path="/var/lib/kubelet/pods/f2828292-0d7e-420d-a5da-ff13c0bdcfa9/volumes"
Oct 30 03:49:09.233566 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:09.233420    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6c5ceef3-a9de-4329-ac29-7d7430c5f03d","Type":"ContainerStarted","Data":"0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6"}
Oct 30 03:49:09.233566 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:09.233494    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6c5ceef3-a9de-4329-ac29-7d7430c5f03d","Type":"ContainerStarted","Data":"baa12ec055035bd380ffd65bf26f49951ebe67ff772ad02bf0273693587ecf9c"}
Oct 30 03:49:09.284032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:09.283986    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.283973214 podStartE2EDuration="1.283973214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:09.281440363 +0000 UTC m=+4028.288030617" watchObservedRunningTime="2026-10-30 03:49:09.283973214 +0000 UTC m=+4028.290563467"
Oct 30 03:49:11.676982 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.676948    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"]
Oct 30 03:49:11.677425 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.677196    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh" podUID="f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" containerName="kube-rbac-proxy" containerID="cri-o://39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289" gracePeriod=30
Oct 30 03:49:11.696563 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.696540    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"]
Oct 30 03:49:11.696768 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.696749    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerName="kube-rbac-proxy" containerID="cri-o://3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315" gracePeriod=30
Oct 30 03:49:11.696827 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.696762    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerName="prom-label-proxy" containerID="cri-o://2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7" gracePeriod=30
Oct 30 03:49:11.716039 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.715908    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"]
Oct 30 03:49:11.716114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.716090    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" containerName="kube-rbac-proxy" containerID="cri-o://00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2" gracePeriod=30
Oct 30 03:49:11.716114 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.716100    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" containerName="prom-label-proxy" containerID="cri-o://f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498" gracePeriod=30
Oct 30 03:49:11.765816 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.765782    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh" podUID="f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" containerName="kube-rbac-proxy" probeResult="failure" output="dial tcp 10.133.0.87:8443: connect: connection refused"
Oct 30 03:49:11.963158 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:11.963130    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:49:12.001840 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.001816    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:49:12.011293 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.011265    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:49:12.043287 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.043260    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-client-ca\") pod \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") "
Oct 30 03:49:12.043430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.043298    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-client-cert\") pod \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") "
Oct 30 03:49:12.043430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.043350    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c2ad8a-19bf-4d26-9b15-31b369395aed-tls\" (UniqueName: \"kubernetes.io/secret/b7c2ad8a-19bf-4d26-9b15-31b369395aed-tls\") pod \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") "
Oct 30 03:49:12.043430 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.043406    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-ca\") pod \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") "
Oct 30 03:49:12.043637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.043434    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-rbac-proxy-config\") pod \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") "
Oct 30 03:49:12.043637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.043504    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c2ad8a-19bf-4d26-9b15-31b369395aed-kube-api-access-jdcm2\" (UniqueName: \"kubernetes.io/projected/b7c2ad8a-19bf-4d26-9b15-31b369395aed-kube-api-access-jdcm2\") pod \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") "
Oct 30 03:49:12.043952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.043709    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-client-ca" pod "b7c2ad8a-19bf-4d26-9b15-31b369395aed" (UID: "b7c2ad8a-19bf-4d26-9b15-31b369395aed"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:12.044202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.044181    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-config\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-config\") pod \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") "
Oct 30 03:49:12.044352 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.044337    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-api-access-7bzbh\" (UniqueName: \"kubernetes.io/projected/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-api-access-7bzbh\") pod \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") "
Oct 30 03:49:12.044508 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.044492    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-prometheus-tls-ca\") pod \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\" (UID: \"b7c2ad8a-19bf-4d26-9b15-31b369395aed\") "
Oct 30 03:49:12.044668 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.044637    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-tls\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-tls\") pod \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\" (UID: \"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07\") "
Oct 30 03:49:12.045032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.045010    4334 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-client-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.045630 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.045295    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-ca" pod "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" (UID: "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:12.045630 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.045594    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-prometheus-tls-ca" pod "b7c2ad8a-19bf-4d26-9b15-31b369395aed" (UID: "b7c2ad8a-19bf-4d26-9b15-31b369395aed"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:12.046205 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.046163    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c2ad8a-19bf-4d26-9b15-31b369395aed-tls" pod "b7c2ad8a-19bf-4d26-9b15-31b369395aed" (UID: "b7c2ad8a-19bf-4d26-9b15-31b369395aed"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:12.047255 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.047231    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c2ad8a-19bf-4d26-9b15-31b369395aed-kube-api-access-jdcm2" pod "b7c2ad8a-19bf-4d26-9b15-31b369395aed" (UID: "b7c2ad8a-19bf-4d26-9b15-31b369395aed"). InnerVolumeSpecName "kube-api-access-jdcm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:12.047773 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.047745    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-tls" pod "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" (UID: "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:12.047861 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.047783    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-client-cert" pod "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" (UID: "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:12.048682 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.048655    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-config" pod "b7c2ad8a-19bf-4d26-9b15-31b369395aed" (UID: "b7c2ad8a-19bf-4d26-9b15-31b369395aed"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:12.049505 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.049465    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-rbac-proxy-config" pod "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" (UID: "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:12.050365 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.050340    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-api-access-7bzbh" pod "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" (UID: "f4e7b6a6-ef32-4784-b89c-77eaaaa84d07"). InnerVolumeSpecName "kube-api-access-7bzbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:12.145857 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.145827    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ab7a71-1334-4148-a65f-f7a703c70558-kube-api-access-ckfgp\" (UniqueName: \"kubernetes.io/projected/90ab7a71-1334-4148-a65f-f7a703c70558-kube-api-access-ckfgp\") pod \"90ab7a71-1334-4148-a65f-f7a703c70558\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") "
Oct 30 03:49:12.146021 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.145896    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls\" (UniqueName: \"kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls\") pod \"90ab7a71-1334-4148-a65f-f7a703c70558\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") "
Oct 30 03:49:12.146021 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.145965    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ab7a71-1334-4148-a65f-f7a703c70558-config\" (UniqueName: \"kubernetes.io/configmap/90ab7a71-1334-4148-a65f-f7a703c70558-config\") pod \"90ab7a71-1334-4148-a65f-f7a703c70558\" (UID: \"90ab7a71-1334-4148-a65f-f7a703c70558\") "
Oct 30 03:49:12.146338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146275    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-prometheus-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146300    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146313    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146328    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b7c2ad8a-19bf-4d26-9b15-31b369395aed-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146341    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-prometheus-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146623 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146354    4334 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-rbac-proxy-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146623 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146367    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcm2\" (UniqueName: \"kubernetes.io/projected/b7c2ad8a-19bf-4d26-9b15-31b369395aed-kube-api-access-jdcm2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146623 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146381    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7c2ad8a-19bf-4d26-9b15-31b369395aed-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146623 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146394    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bzbh\" (UniqueName: \"kubernetes.io/projected/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07-kube-api-access-7bzbh\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.146837 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.146807    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ab7a71-1334-4148-a65f-f7a703c70558-config" pod "90ab7a71-1334-4148-a65f-f7a703c70558" (UID: "90ab7a71-1334-4148-a65f-f7a703c70558"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:12.151285 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.151261    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ab7a71-1334-4148-a65f-f7a703c70558-kube-api-access-ckfgp" pod "90ab7a71-1334-4148-a65f-f7a703c70558" (UID: "90ab7a71-1334-4148-a65f-f7a703c70558"). InnerVolumeSpecName "kube-api-access-ckfgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:12.153569 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.153535    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls" pod "90ab7a71-1334-4148-a65f-f7a703c70558" (UID: "90ab7a71-1334-4148-a65f-f7a703c70558"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:12.183480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.183416    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:49:12.183721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.183675    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="alertmanager" containerID="cri-o://1eb61eb42097b219ca45152a04c7822723f338df4b74a0d2c221e428b1cb64cb" gracePeriod=120
Oct 30 03:49:12.183854 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.183723    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="config-reloader" containerID="cri-o://81dc8ae1ccc783a6f51a083458e381ad26dd9acc2538a6ab13cdb4ddf105ef37" gracePeriod=120
Oct 30 03:49:12.196677 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.196614    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:49:12.196914 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.196895    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="prometheus" containerID="cri-o://5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db" gracePeriod=600
Oct 30 03:49:12.197012 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.196936    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="thanos-sidecar" containerID="cri-o://017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c" gracePeriod=600
Oct 30 03:49:12.197077 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.197037    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="config-reloader" containerID="cri-o://3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce" gracePeriod=600
Oct 30 03:49:12.246934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.246901    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfgp\" (UniqueName: \"kubernetes.io/projected/90ab7a71-1334-4148-a65f-f7a703c70558-kube-api-access-ckfgp\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.246934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.246934    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/90ab7a71-1334-4148-a65f-f7a703c70558-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.247146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.246950    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/90ab7a71-1334-4148-a65f-f7a703c70558-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:12.255044 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.255021    4334 generic.go:356] "Generic (PLEG): container finished" podID="f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" containerID="39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289" exitCode=0
Oct 30 03:49:12.255151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.255062    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"
Oct 30 03:49:12.255151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.255097    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh" event={"ID":"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07","Type":"ContainerDied","Data":"39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289"}
Oct 30 03:49:12.255151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.255144    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh" event={"ID":"f4e7b6a6-ef32-4784-b89c-77eaaaa84d07","Type":"ContainerDied","Data":"c565e45c73743bdec7199e25f056800289117a99d0cb30374688dd41cb31d8f9"}
Oct 30 03:49:12.255286 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.255174    4334 scope.go:122] "RemoveContainer" containerID="39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289"
Oct 30 03:49:12.256900 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.256879    4334 generic.go:356] "Generic (PLEG): container finished" podID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerID="2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7" exitCode=0
Oct 30 03:49:12.256900 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.256900    4334 generic.go:356] "Generic (PLEG): container finished" podID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerID="3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315" exitCode=0
Oct 30 03:49:12.257071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.256954    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"
Oct 30 03:49:12.257071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.256977    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" event={"ID":"b7c2ad8a-19bf-4d26-9b15-31b369395aed","Type":"ContainerDied","Data":"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7"}
Oct 30 03:49:12.257071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.257027    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" event={"ID":"b7c2ad8a-19bf-4d26-9b15-31b369395aed","Type":"ContainerDied","Data":"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315"}
Oct 30 03:49:12.257071 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.257052    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz" event={"ID":"b7c2ad8a-19bf-4d26-9b15-31b369395aed","Type":"ContainerDied","Data":"cd7a4a508b72b847afc1169e4c496c4ac1078e502017eb2388cdd3aa9d1a3075"}
Oct 30 03:49:12.258735 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.258711    4334 generic.go:356] "Generic (PLEG): container finished" podID="90ab7a71-1334-4148-a65f-f7a703c70558" containerID="f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498" exitCode=0
Oct 30 03:49:12.258735 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.258734    4334 generic.go:356] "Generic (PLEG): container finished" podID="90ab7a71-1334-4148-a65f-f7a703c70558" containerID="00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2" exitCode=0
Oct 30 03:49:12.258886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.258754    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"
Oct 30 03:49:12.258886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.258765    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" event={"ID":"90ab7a71-1334-4148-a65f-f7a703c70558","Type":"ContainerDied","Data":"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498"}
Oct 30 03:49:12.258886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.258807    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" event={"ID":"90ab7a71-1334-4148-a65f-f7a703c70558","Type":"ContainerDied","Data":"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2"}
Oct 30 03:49:12.258886 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.258835    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26" event={"ID":"90ab7a71-1334-4148-a65f-f7a703c70558","Type":"ContainerDied","Data":"0ce0bd59e899072ac1df8479a5fd4f66d6540a2f5c522015e0ae7aea72638eda"}
Oct 30 03:49:12.318361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.318333    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"]
Oct 30 03:49:12.322701 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.322678    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-cvdhz"]
Oct 30 03:49:12.359805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.359781    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"]
Oct 30 03:49:12.362782 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.362761    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v5bfh"]
Oct 30 03:49:12.371644 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.371624    4334 scope.go:122] "RemoveContainer" containerID="39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289"
Oct 30 03:49:12.372032 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:12.372013    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289\": container with ID starting with 39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289 not found: ID does not exist" containerID="39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289"
Oct 30 03:49:12.372101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.372041    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289"} err="failed to get container status \"39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289\": rpc error: code = NotFound desc = could not find container \"39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289\": container with ID starting with 39a9a07d5a20171a603403ee270f2a05da88704111f21bb72eab33e38e905289 not found: ID does not exist"
Oct 30 03:49:12.372101 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.372062    4334 scope.go:122] "RemoveContainer" containerID="2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7"
Oct 30 03:49:12.380580 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.380558    4334 scope.go:122] "RemoveContainer" containerID="3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315"
Oct 30 03:49:12.388012 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.387992    4334 scope.go:122] "RemoveContainer" containerID="2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7"
Oct 30 03:49:12.388277 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:12.388259    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7\": container with ID starting with 2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7 not found: ID does not exist" containerID="2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7"
Oct 30 03:49:12.388326 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.388290    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7"} err="failed to get container status \"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7\": rpc error: code = NotFound desc = could not find container \"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7\": container with ID starting with 2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7 not found: ID does not exist"
Oct 30 03:49:12.388326 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.388308    4334 scope.go:122] "RemoveContainer" containerID="3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315"
Oct 30 03:49:12.388688 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:12.388659    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315\": container with ID starting with 3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315 not found: ID does not exist" containerID="3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315"
Oct 30 03:49:12.388749 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.388695    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315"} err="failed to get container status \"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315\": rpc error: code = NotFound desc = could not find container \"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315\": container with ID starting with 3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315 not found: ID does not exist"
Oct 30 03:49:12.388749 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.388711    4334 scope.go:122] "RemoveContainer" containerID="2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7"
Oct 30 03:49:12.389001 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.388981    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7"} err="failed to get container status \"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7\": rpc error: code = NotFound desc = could not find container \"2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7\": container with ID starting with 2a768c88b505b18f6e3578d1bc89a342aba0fe85e2092e3cf9bb6506b2e183e7 not found: ID does not exist"
Oct 30 03:49:12.389001 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.389001    4334 scope.go:122] "RemoveContainer" containerID="3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315"
Oct 30 03:49:12.389382 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.389349    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315"} err="failed to get container status \"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315\": rpc error: code = NotFound desc = could not find container \"3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315\": container with ID starting with 3223e1d2334449702d0bb13f4cad5fcaaf8d762e0189076ca3834fbd30e69315 not found: ID does not exist"
Oct 30 03:49:12.389501 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.389382    4334 scope.go:122] "RemoveContainer" containerID="f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498"
Oct 30 03:49:12.397234 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.397211    4334 scope.go:122] "RemoveContainer" containerID="00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2"
Oct 30 03:49:12.402481 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.402252    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"]
Oct 30 03:49:12.405423 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.405407    4334 scope.go:122] "RemoveContainer" containerID="f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498"
Oct 30 03:49:12.405703 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:12.405680    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498\": container with ID starting with f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498 not found: ID does not exist" containerID="f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498"
Oct 30 03:49:12.405766 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.405714    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498"} err="failed to get container status \"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498\": rpc error: code = NotFound desc = could not find container \"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498\": container with ID starting with f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498 not found: ID does not exist"
Oct 30 03:49:12.405766 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.405737    4334 scope.go:122] "RemoveContainer" containerID="00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2"
Oct 30 03:49:12.405973 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:12.405954    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2\": container with ID starting with 00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2 not found: ID does not exist" containerID="00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2"
Oct 30 03:49:12.406022 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.405980    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2"} err="failed to get container status \"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2\": rpc error: code = NotFound desc = could not find container \"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2\": container with ID starting with 00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2 not found: ID does not exist"
Oct 30 03:49:12.406022 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.405995    4334 scope.go:122] "RemoveContainer" containerID="f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498"
Oct 30 03:49:12.406202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.406184    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-sbk26"]
Oct 30 03:49:12.406273 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.406197    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498"} err="failed to get container status \"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498\": rpc error: code = NotFound desc = could not find container \"f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498\": container with ID starting with f674094188e46bda2bde0d965705870a5524652872568ac99600047616a98498 not found: ID does not exist"
Oct 30 03:49:12.406273 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.406223    4334 scope.go:122] "RemoveContainer" containerID="00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2"
Oct 30 03:49:12.406545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.406523    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2"} err="failed to get container status \"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2\": rpc error: code = NotFound desc = could not find container \"00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2\": container with ID starting with 00e2bdd9eaf79f5b8820d979d60a489040cd4e46ced7d2d1d3254cc6846bf3b2 not found: ID does not exist"
Oct 30 03:49:12.896476 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.896408    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:12.896940 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.896679    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="6c5ceef3-a9de-4329-ac29-7d7430c5f03d" containerName="otc-container" containerID="cri-o://0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6" gracePeriod=30
Oct 30 03:49:12.915191 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:12.915155    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:49:13.150791 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.150767    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:13.164003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.159837    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" path="/var/lib/kubelet/pods/90ab7a71-1334-4148-a65f-f7a703c70558/volumes"
Oct 30 03:49:13.164003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.160485    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" path="/var/lib/kubelet/pods/b7c2ad8a-19bf-4d26-9b15-31b369395aed/volumes"
Oct 30 03:49:13.164003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.161087    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" path="/var/lib/kubelet/pods/f4e7b6a6-ef32-4784-b89c-77eaaaa84d07/volumes"
Oct 30 03:49:13.256653 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.256621    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-kube-api-access-6fbbb\" (UniqueName: \"kubernetes.io/projected/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-kube-api-access-6fbbb\") pod \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") "
Oct 30 03:49:13.256787 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.256665    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-data-science-collector-ta-client-cert\") pod \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") "
Oct 30 03:49:13.256787 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.256719    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-monitoring-tls-certs\") pod \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") "
Oct 30 03:49:13.256787 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.256782    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-tls-certs\") pod \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") "
Oct 30 03:49:13.256928 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.256807    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otlp-tls-certs\") pod \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") "
Oct 30 03:49:13.256928 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.256848    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otc-internal\") pod \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\" (UID: \"6c5ceef3-a9de-4329-ac29-7d7430c5f03d\") "
Oct 30 03:49:13.257324 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.257283    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otc-internal" pod "6c5ceef3-a9de-4329-ac29-7d7430c5f03d" (UID: "6c5ceef3-a9de-4329-ac29-7d7430c5f03d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:13.258858 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.258826    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-data-science-collector-ta-client-cert" pod "6c5ceef3-a9de-4329-ac29-7d7430c5f03d" (UID: "6c5ceef3-a9de-4329-ac29-7d7430c5f03d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.258998 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.258973    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-monitoring-tls-certs" pod "6c5ceef3-a9de-4329-ac29-7d7430c5f03d" (UID: "6c5ceef3-a9de-4329-ac29-7d7430c5f03d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.259152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.259124    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otlp-tls-certs" pod "6c5ceef3-a9de-4329-ac29-7d7430c5f03d" (UID: "6c5ceef3-a9de-4329-ac29-7d7430c5f03d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.259225 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.259197    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-tls-certs" pod "6c5ceef3-a9de-4329-ac29-7d7430c5f03d" (UID: "6c5ceef3-a9de-4329-ac29-7d7430c5f03d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.259303 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.259281    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-kube-api-access-6fbbb" pod "6c5ceef3-a9de-4329-ac29-7d7430c5f03d" (UID: "6c5ceef3-a9de-4329-ac29-7d7430c5f03d"). InnerVolumeSpecName "kube-api-access-6fbbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:13.268202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.268145    4334 generic.go:356] "Generic (PLEG): container finished" podID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerID="017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c" exitCode=0
Oct 30 03:49:13.268202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.268164    4334 generic.go:356] "Generic (PLEG): container finished" podID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerID="3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce" exitCode=0
Oct 30 03:49:13.268330 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.268222    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerDied","Data":"017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c"}
Oct 30 03:49:13.268330 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.268283    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerDied","Data":"3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce"}
Oct 30 03:49:13.269390 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.269369    4334 generic.go:356] "Generic (PLEG): container finished" podID="6c5ceef3-a9de-4329-ac29-7d7430c5f03d" containerID="0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6" exitCode=0
Oct 30 03:49:13.269500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.269420    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:49:13.269500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.269476    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6c5ceef3-a9de-4329-ac29-7d7430c5f03d","Type":"ContainerDied","Data":"0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6"}
Oct 30 03:49:13.269582 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.269528    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6c5ceef3-a9de-4329-ac29-7d7430c5f03d","Type":"ContainerDied","Data":"baa12ec055035bd380ffd65bf26f49951ebe67ff772ad02bf0273693587ecf9c"}
Oct 30 03:49:13.269582 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.269557    4334 scope.go:122] "RemoveContainer" containerID="0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6"
Oct 30 03:49:13.273045 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.273021    4334 generic.go:356] "Generic (PLEG): container finished" podID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerID="81dc8ae1ccc783a6f51a083458e381ad26dd9acc2538a6ab13cdb4ddf105ef37" exitCode=0
Oct 30 03:49:13.273127 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.273061    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6849b93c-0a73-448c-9021-d548ecfd2ba1","Type":"ContainerDied","Data":"81dc8ae1ccc783a6f51a083458e381ad26dd9acc2538a6ab13cdb4ddf105ef37"}
Oct 30 03:49:13.279935 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.279918    4334 scope.go:122] "RemoveContainer" containerID="0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6"
Oct 30 03:49:13.280214 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:13.280194    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6\": container with ID starting with 0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6 not found: ID does not exist" containerID="0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6"
Oct 30 03:49:13.280296 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.280224    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6"} err="failed to get container status \"0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6\": rpc error: code = NotFound desc = could not find container \"0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6\": container with ID starting with 0e2147876538e3c9fdaa4f0dee7363c135a51ce01a0441d17e4345f6cc5d73e6 not found: ID does not exist"
Oct 30 03:49:13.322747 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.322723    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:13.326139 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.326118    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:49:13.358479 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.358460    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fbbb\" (UniqueName: \"kubernetes.io/projected/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-kube-api-access-6fbbb\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.358553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.358481    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.358553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.358492    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.358553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.358501    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.358553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.358510    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.358553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.358518    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c5ceef3-a9de-4329-ac29-7d7430c5f03d-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.752870 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.752846    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:13.863503 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863471    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.863503 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863504    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-tls-assets\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-tls-assets\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.863759 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863530    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-kube-api-access-qb8xb\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-kube-api-access-qb8xb\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.863759 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863568    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.863759 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863607    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.863759 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863704    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.863759 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863733    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.864030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863765    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.864030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863802    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config-out\" (UniqueName: \"kubernetes.io/empty-dir/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config-out\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.864030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863822    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-thanos-prometheus-http-client-file\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.864030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863850    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.864030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863886    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-secret-prometheus-web-tls-ca\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.864030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.863942    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\" (UID: \"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18\") "
Oct 30 03:49:13.865146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.865113    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-0" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:13.865502 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.865473    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-1" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:13.865965 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.865919    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-2" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:13.869133 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.869007    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-secret-prometheus-web-tls-ca" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.869678 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.869651    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-tls-assets" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:13.869815 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.869751    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.869815 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.869808    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.869972 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.869885    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.870785 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.870762    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-thanos-prometheus-http-client-file" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.871763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.871383    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config-out" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:49:13.874241 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.874211    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-kube-api-access-qb8xb" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "kube-api-access-qb8xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:13.874648 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.874616    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:49:13.910483 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.910423    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config" pod "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" (UID: "de58ed3d-7cb5-4588-9f3e-69f0cbc2be18"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:13.964959 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.964931    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.964959 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.964960    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.964976    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.964996    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965010    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965025    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965040    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965053    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8xb\" (UniqueName: \"kubernetes.io/projected/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-kube-api-access-qb8xb\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965064    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965079    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965117    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965132    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.965202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.965147    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:13.981570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:13.981551    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:49:14.065844 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.065814    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\""
Oct 30 03:49:14.279793 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.279766    4334 generic.go:356] "Generic (PLEG): container finished" podID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerID="5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db" exitCode=0
Oct 30 03:49:14.279944 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.279830    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:49:14.279944 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.279836    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerDied","Data":"5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db"}
Oct 30 03:49:14.279944 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.279899    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"de58ed3d-7cb5-4588-9f3e-69f0cbc2be18","Type":"ContainerDied","Data":"e205f6f1b98a827324ff749f8af88cf387b3d92b0e0c33fd1d9899fa2af77c85"}
Oct 30 03:49:14.279944 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.279936    4334 scope.go:122] "RemoveContainer" containerID="017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c"
Oct 30 03:49:14.288542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.288525    4334 scope.go:122] "RemoveContainer" containerID="3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce"
Oct 30 03:49:14.296775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.296750    4334 scope.go:122] "RemoveContainer" containerID="5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db"
Oct 30 03:49:14.304491 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.304473    4334 scope.go:122] "RemoveContainer" containerID="c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39"
Oct 30 03:49:14.312168 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.312153    4334 scope.go:122] "RemoveContainer" containerID="017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c"
Oct 30 03:49:14.312463 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:14.312427    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c\": container with ID starting with 017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c not found: ID does not exist" containerID="017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c"
Oct 30 03:49:14.312526 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.312465    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c"} err="failed to get container status \"017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c\": rpc error: code = NotFound desc = could not find container \"017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c\": container with ID starting with 017bccc6d02b1a81aa8354d4518c965bfdc306fadeab506acd30eead5238946c not found: ID does not exist"
Oct 30 03:49:14.312526 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.312483    4334 scope.go:122] "RemoveContainer" containerID="3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce"
Oct 30 03:49:14.312785 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:14.312762    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce\": container with ID starting with 3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce not found: ID does not exist" containerID="3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce"
Oct 30 03:49:14.312828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.312794    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce"} err="failed to get container status \"3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce\": rpc error: code = NotFound desc = could not find container \"3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce\": container with ID starting with 3fa4e490cfd791937772ce325adb1fb15128b8dfbfddae939bb6c4be818614ce not found: ID does not exist"
Oct 30 03:49:14.312828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.312818    4334 scope.go:122] "RemoveContainer" containerID="5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db"
Oct 30 03:49:14.313065 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:14.313043    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db\": container with ID starting with 5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db not found: ID does not exist" containerID="5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db"
Oct 30 03:49:14.313112 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.313075    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db"} err="failed to get container status \"5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db\": rpc error: code = NotFound desc = could not find container \"5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db\": container with ID starting with 5112f9992a011a104b8f02db50b7edf5c575bba264b33e483fbbb19d527bb5db not found: ID does not exist"
Oct 30 03:49:14.313112 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.313089    4334 scope.go:122] "RemoveContainer" containerID="c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39"
Oct 30 03:49:14.313314 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:14.313298    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39\": container with ID starting with c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39 not found: ID does not exist" containerID="c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39"
Oct 30 03:49:14.313363 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.313317    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39"} err="failed to get container status \"c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39\": rpc error: code = NotFound desc = could not find container \"c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39\": container with ID starting with c0345a12c412a658b0576905fcfd305603080c5e73ced30d7f7d873bd2443f39 not found: ID does not exist"
Oct 30 03:49:14.335914 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.335891    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:49:14.340096 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.340079    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:49:14.810833 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:14.810796    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.91:9093/-/ready\": dial tcp 10.133.0.91:9093: connect: connection refused"
Oct 30 03:49:15.158976 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:15.158948    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5ceef3-a9de-4329-ac29-7d7430c5f03d" path="/var/lib/kubelet/pods/6c5ceef3-a9de-4329-ac29-7d7430c5f03d/volumes"
Oct 30 03:49:15.159376 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:15.159364    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" path="/var/lib/kubelet/pods/de58ed3d-7cb5-4588-9f3e-69f0cbc2be18/volumes"
Oct 30 03:49:19.514845 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.514808    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"]
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515102    4334 cpu_manager.go:414] "containerMap: removing container" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="prometheus"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515112    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="prometheus"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515128    4334 cpu_manager.go:414] "containerMap: removing container" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="config-reloader"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515133    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="config-reloader"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515150    4334 cpu_manager.go:414] "containerMap: removing container" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="thanos-sidecar"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515155    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="thanos-sidecar"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515160    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6c5ceef3-a9de-4329-ac29-7d7430c5f03d" containerName="otc-container"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515165    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5ceef3-a9de-4329-ac29-7d7430c5f03d" containerName="otc-container"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515175    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515179    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515185    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerName="prom-label-proxy"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515190    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerName="prom-label-proxy"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515196    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515201    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515207    4334 cpu_manager.go:414] "containerMap: removing container" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515212    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515219    4334 cpu_manager.go:414] "containerMap: removing container" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="init-config-reloader"
Oct 30 03:49:19.515221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515223    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="init-config-reloader"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515232    4334 cpu_manager.go:414] "containerMap: removing container" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" containerName="prom-label-proxy"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515236    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" containerName="prom-label-proxy"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515281    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="prometheus"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515290    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="config-reloader"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515298    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="de58ed3d-7cb5-4588-9f3e-69f0cbc2be18" containerName="thanos-sidecar"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515304    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5ceef3-a9de-4329-ac29-7d7430c5f03d" containerName="otc-container"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515310    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515314    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c2ad8a-19bf-4d26-9b15-31b369395aed" containerName="prom-label-proxy"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515319    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e7b6a6-ef32-4784-b89c-77eaaaa84d07" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515325    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" containerName="kube-rbac-proxy"
Oct 30 03:49:19.515758 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.515330    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ab7a71-1334-4148-a65f-f7a703c70558" containerName="prom-label-proxy"
Oct 30 03:49:19.520110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.520087    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.535105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.535071    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"]
Oct 30 03:49:19.547480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.547439    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 03:49:19.547570 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.547539    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-zsd8x\""
Oct 30 03:49:19.549198 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.549176    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 03:49:19.606352 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.606324    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/94dadde8-481e-4bc7-827a-0628180c7c20-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.606503 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.606358    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdn9\" (UniqueName: \"kubernetes.io/projected/94dadde8-481e-4bc7-827a-0628180c7c20-kube-api-access-zkdn9\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.606503 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.606485    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94dadde8-481e-4bc7-827a-0628180c7c20-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.707361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.707331    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94dadde8-481e-4bc7-827a-0628180c7c20-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.707495 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.707377    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/94dadde8-481e-4bc7-827a-0628180c7c20-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.707495 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.707396    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdn9\" (UniqueName: \"kubernetes.io/projected/94dadde8-481e-4bc7-827a-0628180c7c20-kube-api-access-zkdn9\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.707979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.707960    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94dadde8-481e-4bc7-827a-0628180c7c20-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.709757 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.709735    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/94dadde8-481e-4bc7-827a-0628180c7c20-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.726710 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.726684    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdn9\" (UniqueName: \"kubernetes.io/projected/94dadde8-481e-4bc7-827a-0628180c7c20-kube-api-access-zkdn9\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jxfns\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:19.811860 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.811797    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.91:9093/-/ready\": dial tcp 10.133.0.91:9093: connect: connection refused"
Oct 30 03:49:19.835205 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:19.835181    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:49:20.130399 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:20.130374    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"]
Oct 30 03:49:20.132231 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:49:20.132201    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94dadde8_481e_4bc7_827a_0628180c7c20.slice/crio-353d4ca8a3b8e423c56a894b38c5bac15d298352345b66c4a068e08bfdc99c61 WatchSource:0}: Error finding container 353d4ca8a3b8e423c56a894b38c5bac15d298352345b66c4a068e08bfdc99c61: Status 404 returned error can't find the container with id 353d4ca8a3b8e423c56a894b38c5bac15d298352345b66c4a068e08bfdc99c61
Oct 30 03:49:20.316083 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:20.316050    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" event={"ID":"94dadde8-481e-4bc7-827a-0628180c7c20","Type":"ContainerStarted","Data":"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c"}
Oct 30 03:49:20.316179 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:20.316120    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" event={"ID":"94dadde8-481e-4bc7-827a-0628180c7c20","Type":"ContainerStarted","Data":"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27"}
Oct 30 03:49:20.316179 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:20.316159    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" event={"ID":"94dadde8-481e-4bc7-827a-0628180c7c20","Type":"ContainerStarted","Data":"353d4ca8a3b8e423c56a894b38c5bac15d298352345b66c4a068e08bfdc99c61"}
Oct 30 03:49:20.355626 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:20.355570    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" podStartSLOduration=1.355553344 podStartE2EDuration="1.355553344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:20.355204812 +0000 UTC m=+4039.361795065" watchObservedRunningTime="2026-10-30 03:49:20.355553344 +0000 UTC m=+4039.362143598"
Oct 30 03:49:20.977093 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:20.977057    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:20.981180 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:20.981155    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:20.987931 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:20.987894    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:49:21.000153 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.000124    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:21.017346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.011038    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 03:49:21.017346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.011162    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:49:21.017346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.011370    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-sg6sg\""
Oct 30 03:49:21.017346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.011481    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:49:21.017346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.011555    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:49:21.017346 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.011603    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 03:49:21.120334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.120298    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.120334 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.120334    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.120637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.120369    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.120637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.120524    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.120637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.120577    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.120637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.120618    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbg4k\" (UniqueName: \"kubernetes.io/projected/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-kube-api-access-rbg4k\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.221659 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.221625    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.221659 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.221659    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.221895 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.221682    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.221895 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.221804    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.221895 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.221853    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.222058 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.221930    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbg4k\" (UniqueName: \"kubernetes.io/projected/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-kube-api-access-rbg4k\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.222889 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.222844    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.224746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.224693    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.224914 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.224889    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.224914 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.224902    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.225232 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.225213    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.249911 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.249861    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbg4k\" (UniqueName: \"kubernetes.io/projected/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-kube-api-access-rbg4k\") pod \"data-science-collector-collector-0\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.296387 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.296364    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:49:21.599801 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.599778    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:49:21.601536 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:49:21.601503    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37152f2_a53d_4103_bc5a_c3aa0ca60c51.slice/crio-20e1d8468078148db7c39823a66dc12d6b0191e0e85fa17c071fe154cdd00fbd WatchSource:0}: Error finding container 20e1d8468078148db7c39823a66dc12d6b0191e0e85fa17c071fe154cdd00fbd: Status 404 returned error can't find the container with id 20e1d8468078148db7c39823a66dc12d6b0191e0e85fa17c071fe154cdd00fbd
Oct 30 03:49:21.603657 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.603629    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:49:21.603719 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:21.603693    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:49:22.331420 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.331384    4334 generic.go:356] "Generic (PLEG): container finished" podID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerID="1eb61eb42097b219ca45152a04c7822723f338df4b74a0d2c221e428b1cb64cb" exitCode=0
Oct 30 03:49:22.331809 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.331484    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6849b93c-0a73-448c-9021-d548ecfd2ba1","Type":"ContainerDied","Data":"1eb61eb42097b219ca45152a04c7822723f338df4b74a0d2c221e428b1cb64cb"}
Oct 30 03:49:22.333243 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.333218    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e37152f2-a53d-4103-bc5a-c3aa0ca60c51","Type":"ContainerStarted","Data":"ab7ffab6c52bad2efee91ef736887b888b7c27a9980c4c65865cbda5c7fea467"}
Oct 30 03:49:22.333383 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.333258    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e37152f2-a53d-4103-bc5a-c3aa0ca60c51","Type":"ContainerStarted","Data":"20e1d8468078148db7c39823a66dc12d6b0191e0e85fa17c071fe154cdd00fbd"}
Oct 30 03:49:22.376683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.376625    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.376610779 podStartE2EDuration="2.376610779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:22.375165868 +0000 UTC m=+4041.381756122" watchObservedRunningTime="2026-10-30 03:49:22.376610779 +0000 UTC m=+4041.383201033"
Oct 30 03:49:22.434620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.434597    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:22.532802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.532729    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-web-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-web-config\") pod \"6849b93c-0a73-448c-9021-d548ecfd2ba1\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") "
Oct 30 03:49:22.532802 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.532772    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-tls-assets\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-tls-assets\") pod \"6849b93c-0a73-448c-9021-d548ecfd2ba1\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") "
Oct 30 03:49:22.532996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.532824    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-volume\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-volume\") pod \"6849b93c-0a73-448c-9021-d548ecfd2ba1\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") "
Oct 30 03:49:22.532996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.532862    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-kube-api-access-ncvpn\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-kube-api-access-ncvpn\") pod \"6849b93c-0a73-448c-9021-d548ecfd2ba1\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") "
Oct 30 03:49:22.532996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.532887    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-alertmanager-data-science-monitoringstack-db\") pod \"6849b93c-0a73-448c-9021-d548ecfd2ba1\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") "
Oct 30 03:49:22.532996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.532947    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-cluster-tls-config\") pod \"6849b93c-0a73-448c-9021-d548ecfd2ba1\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") "
Oct 30 03:49:22.532996 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.532975    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-out\") pod \"6849b93c-0a73-448c-9021-d548ecfd2ba1\" (UID: \"6849b93c-0a73-448c-9021-d548ecfd2ba1\") "
Oct 30 03:49:22.533361 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.533331    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-alertmanager-data-science-monitoringstack-db" pod "6849b93c-0a73-448c-9021-d548ecfd2ba1" (UID: "6849b93c-0a73-448c-9021-d548ecfd2ba1"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:49:22.535353 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.535316    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-tls-assets" pod "6849b93c-0a73-448c-9021-d548ecfd2ba1" (UID: "6849b93c-0a73-448c-9021-d548ecfd2ba1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:22.535488 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.535419    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-volume" pod "6849b93c-0a73-448c-9021-d548ecfd2ba1" (UID: "6849b93c-0a73-448c-9021-d548ecfd2ba1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:22.535646 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.535621    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-kube-api-access-ncvpn" pod "6849b93c-0a73-448c-9021-d548ecfd2ba1" (UID: "6849b93c-0a73-448c-9021-d548ecfd2ba1"). InnerVolumeSpecName "kube-api-access-ncvpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:22.535738 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.535690    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-out" pod "6849b93c-0a73-448c-9021-d548ecfd2ba1" (UID: "6849b93c-0a73-448c-9021-d548ecfd2ba1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:49:22.540045 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.539938    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-cluster-tls-config" pod "6849b93c-0a73-448c-9021-d548ecfd2ba1" (UID: "6849b93c-0a73-448c-9021-d548ecfd2ba1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:22.546610 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.546587    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-web-config" pod "6849b93c-0a73-448c-9021-d548ecfd2ba1" (UID: "6849b93c-0a73-448c-9021-d548ecfd2ba1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:22.633690 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.633668    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.633690 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.633689    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.633808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.633698    4334 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-volume\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.633808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.633707    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvpn\" (UniqueName: \"kubernetes.io/projected/6849b93c-0a73-448c-9021-d548ecfd2ba1-kube-api-access-ncvpn\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.633808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.633716    4334 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.633808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.633726    4334 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6849b93c-0a73-448c-9021-d548ecfd2ba1-cluster-tls-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:22.633808 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:22.633734    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6849b93c-0a73-448c-9021-d548ecfd2ba1-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:49:23.339335 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:23.339298    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6849b93c-0a73-448c-9021-d548ecfd2ba1","Type":"ContainerDied","Data":"5f14e011d97934cd6911d720a0c515e5a7112962e8a22b257eeb7412745961bc"}
Oct 30 03:49:23.339754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:23.339369    4334 scope.go:122] "RemoveContainer" containerID="81dc8ae1ccc783a6f51a083458e381ad26dd9acc2538a6ab13cdb4ddf105ef37"
Oct 30 03:49:23.339754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:23.339483    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:23.348805 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:23.348785    4334 scope.go:122] "RemoveContainer" containerID="1eb61eb42097b219ca45152a04c7822723f338df4b74a0d2c221e428b1cb64cb"
Oct 30 03:49:23.357828 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:23.357813    4334 scope.go:122] "RemoveContainer" containerID="e50474ecc1998acb08bb4d92d132741c508ffff4c7b013a42d456da64902dec9"
Oct 30 03:49:23.407318 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:23.407289    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:49:23.421158 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:23.421105    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:49:24.458313 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458279    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:49:24.458843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458702    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="config-reloader"
Oct 30 03:49:24.458843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458716    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="config-reloader"
Oct 30 03:49:24.458843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458735    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="alertmanager"
Oct 30 03:49:24.458843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458740    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="alertmanager"
Oct 30 03:49:24.458843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458748    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="init-config-reloader"
Oct 30 03:49:24.458843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458752    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="init-config-reloader"
Oct 30 03:49:24.458843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458824    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="config-reloader"
Oct 30 03:49:24.458843 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.458833    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" containerName="alertmanager"
Oct 30 03:49:24.463780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.463759    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.496256 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.496198    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 03:49:24.496256 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.496214    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:49:24.496256 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.496197    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 03:49:24.496519 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.496268    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 03:49:24.496519 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.496285    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-qbcnw\""
Oct 30 03:49:24.497960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.497938    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:49:24.544140 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.544115    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:49:24.548198 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.548178    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.548974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.548953    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.549065 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.548992    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.549065 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.549025    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.549152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.549087    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.549152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.549111    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.549152 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.549127    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m469p\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-kube-api-access-m469p\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.549273 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.549160    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.567814 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.567791    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:49:24.568098 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.568076    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:49:24.570367 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.570346    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:49:24.570501 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.570481    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:49:24.588798 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.588761    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:49:24.588798 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.588779    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:49:24.588798 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.588787    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:49:24.589056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.588977    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:49:24.589056 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.589013    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-5m2cs\""
Oct 30 03:49:24.589160 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.589057    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:49:24.589160 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.589114    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:49:24.650263 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650231    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.650437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650270    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.650437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650297    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.650437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650350    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9527f9c0-f91a-4e27-89c1-9463688a3269-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.650437 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650388    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.650713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650495    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.650713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650524    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.650713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650556    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.650713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650664    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.650713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650689    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.650713 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650712    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.651003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650736    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m469p\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-kube-api-access-m469p\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.651003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650768    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xtf\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-kube-api-access-d4xtf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.651003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650800    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.651003 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.650975    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.651229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.651010    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.651229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.651057    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.651229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.651159    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.651229 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.651197    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.651685 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.651658    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.652897 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.652872    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.653230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.653210    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.653319 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.653305    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.653475 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.653442    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.653563 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.653548    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.670673 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.670649    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m469p\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-kube-api-access-m469p\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:24.751800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.751738    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xtf\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-kube-api-access-d4xtf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.751800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.751765    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.751800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.751792    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.751829    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.751864    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.751885    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.751911    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.751969    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.752008    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9527f9c0-f91a-4e27-89c1-9463688a3269-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752415 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.752240    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752415 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.752280    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752415 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.752306    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752620 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.752567    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.752934 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.752903    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.753030 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.752945    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.754741 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.754698    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.755078 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.755055    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9527f9c0-f91a-4e27-89c1-9463688a3269-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.755237 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.755213    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.755515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.755477    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.755622 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.755514    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.755715 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.755696    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.755825 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.755806    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.755897 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.755881    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.772434 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.772410    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xtf\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-kube-api-access-d4xtf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:24.790286 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:24.790261    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:25.114155 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:25.114129    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:49:25.115889 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:49:25.115864    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3626f7f5_68a5_4296_8ecb_db1479620ed9.slice/crio-164bc072b56013f2d5a00c394da61bccd7ad12b1255bd26bd56401217c22e2ae WatchSource:0}: Error finding container 164bc072b56013f2d5a00c394da61bccd7ad12b1255bd26bd56401217c22e2ae: Status 404 returned error can't find the container with id 164bc072b56013f2d5a00c394da61bccd7ad12b1255bd26bd56401217c22e2ae
Oct 30 03:49:25.159151 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:25.159127    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6849b93c-0a73-448c-9021-d548ecfd2ba1" path="/var/lib/kubelet/pods/6849b93c-0a73-448c-9021-d548ecfd2ba1/volumes"
Oct 30 03:49:25.351145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:25.351107    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3626f7f5-68a5-4296-8ecb-db1479620ed9","Type":"ContainerStarted","Data":"164bc072b56013f2d5a00c394da61bccd7ad12b1255bd26bd56401217c22e2ae"}
Oct 30 03:49:26.357824 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:26.357736    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3626f7f5-68a5-4296-8ecb-db1479620ed9","Type":"ContainerStarted","Data":"36fc73de7fe43fde89b7d995b8b4a9009698270331ecb4b5b9550f54133f5ee6"}
Oct 30 03:49:29.374607 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:29.374573    4334 generic.go:356] "Generic (PLEG): container finished" podID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerID="36fc73de7fe43fde89b7d995b8b4a9009698270331ecb4b5b9550f54133f5ee6" exitCode=0
Oct 30 03:49:29.375016 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:29.374629    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3626f7f5-68a5-4296-8ecb-db1479620ed9","Type":"ContainerDied","Data":"36fc73de7fe43fde89b7d995b8b4a9009698270331ecb4b5b9550f54133f5ee6"}
Oct 30 03:49:30.382142 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:30.382110    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3626f7f5-68a5-4296-8ecb-db1479620ed9","Type":"ContainerStarted","Data":"27b5cfe406ca87a86eb0942d42aaf702fda6943a580bf8419bd065d9c0a54743"}
Oct 30 03:49:30.382530 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:30.382172    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3626f7f5-68a5-4296-8ecb-db1479620ed9","Type":"ContainerStarted","Data":"ac4c2d64eac2a3b13a30022c57fe2b986f924abf41377df1f1fbe835f597465c"}
Oct 30 03:49:30.382530 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:30.382353    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:30.453019 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:30.452974    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.169415902 podStartE2EDuration="6.452962097s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.283546195s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:30.452781579 +0000 UTC m=+4049.459371833" watchObservedRunningTime="2026-10-30 03:49:30.452962097 +0000 UTC m=+4049.459552350"
Oct 30 03:49:34.728674 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:34.728643    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:34.758950 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:34.758926    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:34.829679 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:34.829652    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:34.829794 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:34.829767    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:34.844754 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:34.844724    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") DevicePath \"csi-e364dd4f86bda642cef8d14035ca0bfc3cdafe8ca20f42d905720eb71a97db6d\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:34.949209 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:34.949177    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8c516923ea0339579ebf808cef81b9e0aaf55e5b619e2c6187025fa3b2b51f7f/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:34.976145 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:34.976122    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:35.071637 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:35.071580    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:35.399257 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:35.399230    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:49:35.400389 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:49:35.400358    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9527f9c0_f91a_4e27_89c1_9463688a3269.slice/crio-3d3a33ea532588f11a5c0a1f1bfa42293c3b2cee73eddbf207bedc3b87704bc7 WatchSource:0}: Error finding container 3d3a33ea532588f11a5c0a1f1bfa42293c3b2cee73eddbf207bedc3b87704bc7: Status 404 returned error can't find the container with id 3d3a33ea532588f11a5c0a1f1bfa42293c3b2cee73eddbf207bedc3b87704bc7
Oct 30 03:49:35.407239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:35.407183    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerStarted","Data":"3d3a33ea532588f11a5c0a1f1bfa42293c3b2cee73eddbf207bedc3b87704bc7"}
Oct 30 03:49:36.389374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:36.389342    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:49:36.413497 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:36.413467    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerStarted","Data":"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"}
Oct 30 03:49:39.429433 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:39.429401    4334 generic.go:356] "Generic (PLEG): container finished" podID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerID="60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6" exitCode=0
Oct 30 03:49:39.429818 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:39.429484    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerDied","Data":"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"}
Oct 30 03:49:40.437576 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:40.437545    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerStarted","Data":"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"}
Oct 30 03:49:40.437953 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:40.437608    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerStarted","Data":"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"}
Oct 30 03:49:40.437953 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:40.437633    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerStarted","Data":"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"}
Oct 30 03:49:40.512612 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:40.512564    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=12.927812659 podStartE2EDuration="16.512550533s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584737874s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:40.497843641 +0000 UTC m=+4059.504433895" watchObservedRunningTime="2026-10-30 03:49:40.512550533 +0000 UTC m=+4059.519140786"
Oct 30 03:49:45.072465 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:45.072409    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:50.071841 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:50.071805    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:50.077164 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:50.077141    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:50.491809 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:49:50.491786    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:49:50.978806 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:49:50.978775    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:50:20.977760 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:50:20.977683    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:50:36.286157 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.286122    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"]
Oct 30 03:50:36.286740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.286337    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" containerName="kube-rbac-proxy" containerID="cri-o://10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27" gracePeriod=30
Oct 30 03:50:36.286740 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.286365    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" containerName="prom-label-proxy" containerID="cri-o://f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c" gracePeriod=30
Oct 30 03:50:36.533685 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.533660    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:50:36.634830 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.634762    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94dadde8-481e-4bc7-827a-0628180c7c20-kube-api-access-zkdn9\" (UniqueName: \"kubernetes.io/projected/94dadde8-481e-4bc7-827a-0628180c7c20-kube-api-access-zkdn9\") pod \"94dadde8-481e-4bc7-827a-0628180c7c20\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") "
Oct 30 03:50:36.640051 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.640015    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94dadde8-481e-4bc7-827a-0628180c7c20-kube-api-access-zkdn9" pod "94dadde8-481e-4bc7-827a-0628180c7c20" (UID: "94dadde8-481e-4bc7-827a-0628180c7c20"). InnerVolumeSpecName "kube-api-access-zkdn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:36.709981 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.709942    4334 generic.go:356] "Generic (PLEG): container finished" podID="94dadde8-481e-4bc7-827a-0628180c7c20" containerID="f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c" exitCode=0
Oct 30 03:50:36.709981 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.709978    4334 generic.go:356] "Generic (PLEG): container finished" podID="94dadde8-481e-4bc7-827a-0628180c7c20" containerID="10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27" exitCode=0
Oct 30 03:50:36.710230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.710062    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" event={"ID":"94dadde8-481e-4bc7-827a-0628180c7c20","Type":"ContainerDied","Data":"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c"}
Oct 30 03:50:36.710230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.710108    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" event={"ID":"94dadde8-481e-4bc7-827a-0628180c7c20","Type":"ContainerDied","Data":"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27"}
Oct 30 03:50:36.710230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.710132    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns" event={"ID":"94dadde8-481e-4bc7-827a-0628180c7c20","Type":"ContainerDied","Data":"353d4ca8a3b8e423c56a894b38c5bac15d298352345b66c4a068e08bfdc99c61"}
Oct 30 03:50:36.710230 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.710158    4334 scope.go:122] "RemoveContainer" containerID="f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c"
Oct 30 03:50:36.710525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.710297    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"
Oct 30 03:50:36.721343 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.721230    4334 scope.go:122] "RemoveContainer" containerID="10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27"
Oct 30 03:50:36.730270 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.730248    4334 scope.go:122] "RemoveContainer" containerID="f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c"
Oct 30 03:50:36.730647 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:50:36.730616    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c\": container with ID starting with f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c not found: ID does not exist" containerID="f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c"
Oct 30 03:50:36.730744 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.730657    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c"} err="failed to get container status \"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c\": rpc error: code = NotFound desc = could not find container \"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c\": container with ID starting with f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c not found: ID does not exist"
Oct 30 03:50:36.730744 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.730676    4334 scope.go:122] "RemoveContainer" containerID="10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27"
Oct 30 03:50:36.731015 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:50:36.730993    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27\": container with ID starting with 10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27 not found: ID does not exist" containerID="10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27"
Oct 30 03:50:36.731105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.731020    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27"} err="failed to get container status \"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27\": rpc error: code = NotFound desc = could not find container \"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27\": container with ID starting with 10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27 not found: ID does not exist"
Oct 30 03:50:36.731105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.731036    4334 scope.go:122] "RemoveContainer" containerID="f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c"
Oct 30 03:50:36.731332 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.731314    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c"} err="failed to get container status \"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c\": rpc error: code = NotFound desc = could not find container \"f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c\": container with ID starting with f4b13b6e887da3d762596c5ce8212bf8c36ddbe075d3e4b81774e32f768d5b2c not found: ID does not exist"
Oct 30 03:50:36.731332 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.731331    4334 scope.go:122] "RemoveContainer" containerID="10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27"
Oct 30 03:50:36.731667 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.731628    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27"} err="failed to get container status \"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27\": rpc error: code = NotFound desc = could not find container \"10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27\": container with ID starting with 10ebca5363ef6475db3fcb5785ba65b5dbc77f2d0be9c09da51ce9043c345f27 not found: ID does not exist"
Oct 30 03:50:36.737543 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.736600    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94dadde8-481e-4bc7-827a-0628180c7c20-config\" (UniqueName: \"kubernetes.io/configmap/94dadde8-481e-4bc7-827a-0628180c7c20-config\") pod \"94dadde8-481e-4bc7-827a-0628180c7c20\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") "
Oct 30 03:50:36.737543 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.736656    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94dadde8-481e-4bc7-827a-0628180c7c20-tls\" (UniqueName: \"kubernetes.io/secret/94dadde8-481e-4bc7-827a-0628180c7c20-tls\") pod \"94dadde8-481e-4bc7-827a-0628180c7c20\" (UID: \"94dadde8-481e-4bc7-827a-0628180c7c20\") "
Oct 30 03:50:36.737543 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.737180    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdn9\" (UniqueName: \"kubernetes.io/projected/94dadde8-481e-4bc7-827a-0628180c7c20-kube-api-access-zkdn9\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.742110 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.741918    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94dadde8-481e-4bc7-827a-0628180c7c20-tls" pod "94dadde8-481e-4bc7-827a-0628180c7c20" (UID: "94dadde8-481e-4bc7-827a-0628180c7c20"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:36.744195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.742339    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94dadde8-481e-4bc7-827a-0628180c7c20-config" pod "94dadde8-481e-4bc7-827a-0628180c7c20" (UID: "94dadde8-481e-4bc7-827a-0628180c7c20"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:36.744195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.743589    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:50:36.744195 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.743974    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="alertmanager" containerID="cri-o://ac4c2d64eac2a3b13a30022c57fe2b986f924abf41377df1f1fbe835f597465c" gracePeriod=120
Oct 30 03:50:36.744708 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.744341    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="config-reloader" containerID="cri-o://27b5cfe406ca87a86eb0942d42aaf702fda6943a580bf8419bd065d9c0a54743" gracePeriod=120
Oct 30 03:50:36.749538 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.749513    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:50:36.749814 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.749790    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="prometheus" containerID="cri-o://1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574" gracePeriod=600
Oct 30 03:50:36.750032 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.749957    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="thanos-sidecar" containerID="cri-o://6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f" gracePeriod=600
Oct 30 03:50:36.750133 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.750026    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="config-reloader" containerID="cri-o://6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032" gracePeriod=600
Oct 30 03:50:36.838034 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.838003    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94dadde8-481e-4bc7-827a-0628180c7c20-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.838034 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.838027    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/94dadde8-481e-4bc7-827a-0628180c7c20-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:36.998716 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:36.998692    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:50:37.039969 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.039938    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-thanos-prometheus-http-client-file\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.039978    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-secret-prometheus-web-tls-ca\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040007    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9527f9c0-f91a-4e27-89c1-9463688a3269-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9527f9c0-f91a-4e27-89c1-9463688a3269-config-out\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040049    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040071    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040119 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040088    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-kube-api-access-d4xtf\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-kube-api-access-d4xtf\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040152    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-tls-assets\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-tls-assets\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040260    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040293    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040319    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-config\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040384 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040360    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040714 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040396    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040714 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040421    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-1" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:37.040714 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040431    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"9527f9c0-f91a-4e27-89c1-9463688a3269\" (UID: \"9527f9c0-f91a-4e27-89c1-9463688a3269\") "
Oct 30 03:50:37.040714 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.040699    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.041369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.041339    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-0" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:37.041369 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.041361    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-2" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:37.043171 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.043145    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.043275 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.043174    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-thanos-prometheus-http-client-file" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.043275 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.043232    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.043495 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.043440    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-secret-prometheus-web-tls-ca" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.043495 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.043478    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-kube-api-access-d4xtf" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "kube-api-access-d4xtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:37.043634 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.043489    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9527f9c0-f91a-4e27-89c1-9463688a3269-config-out" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:37.043634 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.043498    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-config" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.043985 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.043969    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-tls-assets" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:37.046813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.046794    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "pvc-9689c530-7f91-4584-b28e-8809bed9e799". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:50:37.057194 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.057163    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config" pod "9527f9c0-f91a-4e27-89c1-9463688a3269" (UID: "9527f9c0-f91a-4e27-89c1-9463688a3269"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.086953 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.086925    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"]
Oct 30 03:50:37.091133 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.091108    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jxfns"]
Oct 30 03:50:37.141094 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141063    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141094 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141092    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141108    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9527f9c0-f91a-4e27-89c1-9463688a3269-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141122    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141135    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xtf\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-kube-api-access-d4xtf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141148    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9527f9c0-f91a-4e27-89c1-9463688a3269-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141185    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141199    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141212    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141224    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141238    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9527f9c0-f91a-4e27-89c1-9463688a3269-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.141299 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.141251    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9527f9c0-f91a-4e27-89c1-9463688a3269-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.152048 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.152029    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9689c530-7f91-4584-b28e-8809bed9e799" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87") on node "ip-10-0-133-197.ec2.internal"
Oct 30 03:50:37.158435 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.158411    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" path="/var/lib/kubelet/pods/94dadde8-481e-4bc7-827a-0628180c7c20/volumes"
Oct 30 03:50:37.242540 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.242479    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-e364dd4f86bda642cef8d14035ca0bfc3cdafe8ca20f42d905720eb71a97db6d\""
Oct 30 03:50:37.448053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.448011    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:50:37.448530 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.448320    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="e37152f2-a53d-4103-bc5a-c3aa0ca60c51" containerName="otc-container" containerID="cri-o://ab7ffab6c52bad2efee91ef736887b888b7c27a9980c4c65865cbda5c7fea467" gracePeriod=30
Oct 30 03:50:37.717629 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.717457    4334 generic.go:356] "Generic (PLEG): container finished" podID="e37152f2-a53d-4103-bc5a-c3aa0ca60c51" containerID="ab7ffab6c52bad2efee91ef736887b888b7c27a9980c4c65865cbda5c7fea467" exitCode=0
Oct 30 03:50:37.717629 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.717505    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e37152f2-a53d-4103-bc5a-c3aa0ca60c51","Type":"ContainerDied","Data":"ab7ffab6c52bad2efee91ef736887b888b7c27a9980c4c65865cbda5c7fea467"}
Oct 30 03:50:37.719615 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.719588    4334 generic.go:356] "Generic (PLEG): container finished" podID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerID="27b5cfe406ca87a86eb0942d42aaf702fda6943a580bf8419bd065d9c0a54743" exitCode=0
Oct 30 03:50:37.719615 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.719613    4334 generic.go:356] "Generic (PLEG): container finished" podID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerID="ac4c2d64eac2a3b13a30022c57fe2b986f924abf41377df1f1fbe835f597465c" exitCode=0
Oct 30 03:50:37.719812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.719658    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3626f7f5-68a5-4296-8ecb-db1479620ed9","Type":"ContainerDied","Data":"27b5cfe406ca87a86eb0942d42aaf702fda6943a580bf8419bd065d9c0a54743"}
Oct 30 03:50:37.719812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.719710    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3626f7f5-68a5-4296-8ecb-db1479620ed9","Type":"ContainerDied","Data":"ac4c2d64eac2a3b13a30022c57fe2b986f924abf41377df1f1fbe835f597465c"}
Oct 30 03:50:37.719812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.719738    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3626f7f5-68a5-4296-8ecb-db1479620ed9","Type":"ContainerDied","Data":"164bc072b56013f2d5a00c394da61bccd7ad12b1255bd26bd56401217c22e2ae"}
Oct 30 03:50:37.719812 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.719767    4334 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="164bc072b56013f2d5a00c394da61bccd7ad12b1255bd26bd56401217c22e2ae"
Oct 30 03:50:37.721974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.721950    4334 generic.go:356] "Generic (PLEG): container finished" podID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerID="6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f" exitCode=0
Oct 30 03:50:37.721974 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.721973    4334 generic.go:356] "Generic (PLEG): container finished" podID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerID="6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032" exitCode=0
Oct 30 03:50:37.722100 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.721984    4334 generic.go:356] "Generic (PLEG): container finished" podID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerID="1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574" exitCode=0
Oct 30 03:50:37.722100 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.722008    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerDied","Data":"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"}
Oct 30 03:50:37.722100 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.722078    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerDied","Data":"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"}
Oct 30 03:50:37.722100 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.722013    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:50:37.722266 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.722119    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerDied","Data":"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"}
Oct 30 03:50:37.722266 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.722158    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"9527f9c0-f91a-4e27-89c1-9463688a3269","Type":"ContainerDied","Data":"3d3a33ea532588f11a5c0a1f1bfa42293c3b2cee73eddbf207bedc3b87704bc7"}
Oct 30 03:50:37.722266 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.722172    4334 scope.go:122] "RemoveContainer" containerID="6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"
Oct 30 03:50:37.723008 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.722992    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:50:37.725833 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.725814    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:50:37.730308 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.730292    4334 scope.go:122] "RemoveContainer" containerID="6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"
Oct 30 03:50:37.738017 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.737997    4334 scope.go:122] "RemoveContainer" containerID="1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"
Oct 30 03:50:37.745579 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.745558    4334 scope.go:122] "RemoveContainer" containerID="60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"
Oct 30 03:50:37.746141 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746121    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otc-internal\") pod \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") "
Oct 30 03:50:37.746223 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746188    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-cluster-tls-config\") pod \"3626f7f5-68a5-4296-8ecb-db1479620ed9\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") "
Oct 30 03:50:37.746278 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746228    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-tls-certs\") pod \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") "
Oct 30 03:50:37.746278 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746255    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-monitoring-tls-certs\") pod \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") "
Oct 30 03:50:37.746349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746291    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-out\") pod \"3626f7f5-68a5-4296-8ecb-db1479620ed9\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") "
Oct 30 03:50:37.746349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746321    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otlp-tls-certs\") pod \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") "
Oct 30 03:50:37.746436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746347    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-tls-assets\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-tls-assets\") pod \"3626f7f5-68a5-4296-8ecb-db1479620ed9\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") "
Oct 30 03:50:37.746436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746373    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-web-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-web-config\") pod \"3626f7f5-68a5-4296-8ecb-db1479620ed9\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") "
Oct 30 03:50:37.746436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746409    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-data-science-collector-ta-client-cert\") pod \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") "
Oct 30 03:50:37.746650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746436    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-volume\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-volume\") pod \"3626f7f5-68a5-4296-8ecb-db1479620ed9\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") "
Oct 30 03:50:37.746650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746513    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-alertmanager-data-science-monitoringstack-db\") pod \"3626f7f5-68a5-4296-8ecb-db1479620ed9\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") "
Oct 30 03:50:37.746650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746537    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otc-internal" pod "e37152f2-a53d-4103-bc5a-c3aa0ca60c51" (UID: "e37152f2-a53d-4103-bc5a-c3aa0ca60c51"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:37.746650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746556    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-kube-api-access-rbg4k\" (UniqueName: \"kubernetes.io/projected/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-kube-api-access-rbg4k\") pod \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\" (UID: \"e37152f2-a53d-4103-bc5a-c3aa0ca60c51\") "
Oct 30 03:50:37.746650 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746581    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-kube-api-access-m469p\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-kube-api-access-m469p\") pod \"3626f7f5-68a5-4296-8ecb-db1479620ed9\" (UID: \"3626f7f5-68a5-4296-8ecb-db1479620ed9\") "
Oct 30 03:50:37.746978 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.746790    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.747366 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.747312    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-alertmanager-data-science-monitoringstack-db" pod "3626f7f5-68a5-4296-8ecb-db1479620ed9" (UID: "3626f7f5-68a5-4296-8ecb-db1479620ed9"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:37.749441 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.749411    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-data-science-collector-ta-client-cert" pod "e37152f2-a53d-4103-bc5a-c3aa0ca60c51" (UID: "e37152f2-a53d-4103-bc5a-c3aa0ca60c51"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.749441 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.749427    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-monitoring-tls-certs" pod "e37152f2-a53d-4103-bc5a-c3aa0ca60c51" (UID: "e37152f2-a53d-4103-bc5a-c3aa0ca60c51"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.750910 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.750883    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-kube-api-access-rbg4k" pod "e37152f2-a53d-4103-bc5a-c3aa0ca60c51" (UID: "e37152f2-a53d-4103-bc5a-c3aa0ca60c51"). InnerVolumeSpecName "kube-api-access-rbg4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:37.750910 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.750898    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otlp-tls-certs" pod "e37152f2-a53d-4103-bc5a-c3aa0ca60c51" (UID: "e37152f2-a53d-4103-bc5a-c3aa0ca60c51"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.751064 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.750937    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-tls-certs" pod "e37152f2-a53d-4103-bc5a-c3aa0ca60c51" (UID: "e37152f2-a53d-4103-bc5a-c3aa0ca60c51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.751064 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.750973    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-kube-api-access-m469p" pod "3626f7f5-68a5-4296-8ecb-db1479620ed9" (UID: "3626f7f5-68a5-4296-8ecb-db1479620ed9"). InnerVolumeSpecName "kube-api-access-m469p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:37.751064 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.751023    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-tls-assets" pod "3626f7f5-68a5-4296-8ecb-db1479620ed9" (UID: "3626f7f5-68a5-4296-8ecb-db1479620ed9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:37.752431 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.752405    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-out" pod "3626f7f5-68a5-4296-8ecb-db1479620ed9" (UID: "3626f7f5-68a5-4296-8ecb-db1479620ed9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:37.752534 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.752505    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-volume" pod "3626f7f5-68a5-4296-8ecb-db1479620ed9" (UID: "3626f7f5-68a5-4296-8ecb-db1479620ed9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.753806 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.753782    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-cluster-tls-config" pod "3626f7f5-68a5-4296-8ecb-db1479620ed9" (UID: "3626f7f5-68a5-4296-8ecb-db1479620ed9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.755864 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.755844    4334 scope.go:122] "RemoveContainer" containerID="6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"
Oct 30 03:50:37.756100 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:50:37.756082    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f\": container with ID starting with 6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f not found: ID does not exist" containerID="6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"
Oct 30 03:50:37.756167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.756105    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"} err="failed to get container status \"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f\": rpc error: code = NotFound desc = could not find container \"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f\": container with ID starting with 6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f not found: ID does not exist"
Oct 30 03:50:37.756167 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.756120    4334 scope.go:122] "RemoveContainer" containerID="6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"
Oct 30 03:50:37.756329 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:50:37.756313    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032\": container with ID starting with 6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032 not found: ID does not exist" containerID="6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"
Oct 30 03:50:37.756389 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.756333    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"} err="failed to get container status \"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032\": rpc error: code = NotFound desc = could not find container \"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032\": container with ID starting with 6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032 not found: ID does not exist"
Oct 30 03:50:37.756389 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.756346    4334 scope.go:122] "RemoveContainer" containerID="1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"
Oct 30 03:50:37.756581 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:50:37.756565    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574\": container with ID starting with 1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574 not found: ID does not exist" containerID="1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"
Oct 30 03:50:37.756664 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.756584    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"} err="failed to get container status \"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574\": rpc error: code = NotFound desc = could not find container \"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574\": container with ID starting with 1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574 not found: ID does not exist"
Oct 30 03:50:37.756664 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.756596    4334 scope.go:122] "RemoveContainer" containerID="60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"
Oct 30 03:50:37.756898 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:50:37.756877    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6\": container with ID starting with 60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6 not found: ID does not exist" containerID="60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"
Oct 30 03:50:37.756965 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.756903    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"} err="failed to get container status \"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6\": rpc error: code = NotFound desc = could not find container \"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6\": container with ID starting with 60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6 not found: ID does not exist"
Oct 30 03:50:37.756965 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.756918    4334 scope.go:122] "RemoveContainer" containerID="6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"
Oct 30 03:50:37.757295 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.757276    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"} err="failed to get container status \"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f\": rpc error: code = NotFound desc = could not find container \"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f\": container with ID starting with 6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f not found: ID does not exist"
Oct 30 03:50:37.757358 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.757296    4334 scope.go:122] "RemoveContainer" containerID="6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"
Oct 30 03:50:37.757629 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.757606    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"} err="failed to get container status \"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032\": rpc error: code = NotFound desc = could not find container \"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032\": container with ID starting with 6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032 not found: ID does not exist"
Oct 30 03:50:37.757629 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.757628    4334 scope.go:122] "RemoveContainer" containerID="1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"
Oct 30 03:50:37.757876 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.757854    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"} err="failed to get container status \"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574\": rpc error: code = NotFound desc = could not find container \"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574\": container with ID starting with 1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574 not found: ID does not exist"
Oct 30 03:50:37.757952 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.757877    4334 scope.go:122] "RemoveContainer" containerID="60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"
Oct 30 03:50:37.758188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.758165    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"} err="failed to get container status \"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6\": rpc error: code = NotFound desc = could not find container \"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6\": container with ID starting with 60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6 not found: ID does not exist"
Oct 30 03:50:37.758188 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.758187    4334 scope.go:122] "RemoveContainer" containerID="6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"
Oct 30 03:50:37.758569 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.758437    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f"} err="failed to get container status \"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f\": rpc error: code = NotFound desc = could not find container \"6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f\": container with ID starting with 6f27788d2f645a146c07408dc670996e2796b749ed8d19238df69600e3bead2f not found: ID does not exist"
Oct 30 03:50:37.758569 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.758479    4334 scope.go:122] "RemoveContainer" containerID="6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"
Oct 30 03:50:37.758797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.758770    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032"} err="failed to get container status \"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032\": rpc error: code = NotFound desc = could not find container \"6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032\": container with ID starting with 6b8cb9a3cc6e03548b44e8a7c4c8c26a7fc3536365486cc528dc87ceb38fd032 not found: ID does not exist"
Oct 30 03:50:37.758797 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.758792    4334 scope.go:122] "RemoveContainer" containerID="1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"
Oct 30 03:50:37.759183 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.759086    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574"} err="failed to get container status \"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574\": rpc error: code = NotFound desc = could not find container \"1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574\": container with ID starting with 1f2acf8efd1767a637fb7db0629bc9725fc14b155f98a87a3d78cb8565fa8574 not found: ID does not exist"
Oct 30 03:50:37.759183 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.759115    4334 scope.go:122] "RemoveContainer" containerID="60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"
Oct 30 03:50:37.759401 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.759378    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6"} err="failed to get container status \"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6\": rpc error: code = NotFound desc = could not find container \"60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6\": container with ID starting with 60f760030526b43a39a9db10b27184a585405203e9a9137f331e1694eac7eff6 not found: ID does not exist"
Oct 30 03:50:37.764702 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.764680    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-web-config" pod "3626f7f5-68a5-4296-8ecb-db1479620ed9" (UID: "3626f7f5-68a5-4296-8ecb-db1479620ed9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:37.847342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847256    4334 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847298    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbg4k\" (UniqueName: \"kubernetes.io/projected/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-kube-api-access-rbg4k\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847316    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m469p\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-kube-api-access-m469p\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847342 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847331    4334 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-cluster-tls-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847346    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847359    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847372    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847387    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847400    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3626f7f5-68a5-4296-8ecb-db1479620ed9-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847412    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847426    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e37152f2-a53d-4103-bc5a-c3aa0ca60c51-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.847617 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.847439    4334 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3626f7f5-68a5-4296-8ecb-db1479620ed9-config-volume\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 03:50:37.865548 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.865523    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:50:37.868720 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:37.868699    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:50:38.735364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:38.730006    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"
Oct 30 03:50:38.735364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:38.732371    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:50:38.735364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:38.734555    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e37152f2-a53d-4103-bc5a-c3aa0ca60c51","Type":"ContainerDied","Data":"20e1d8468078148db7c39823a66dc12d6b0191e0e85fa17c071fe154cdd00fbd"}
Oct 30 03:50:38.735364 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:38.734610    4334 scope.go:122] "RemoveContainer" containerID="ab7ffab6c52bad2efee91ef736887b888b7c27a9980c4c65865cbda5c7fea467"
Oct 30 03:50:38.805545 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:38.805520    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:50:38.807130 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:38.807111    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:50:38.843205 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:38.843166    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:50:38.847262 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:38.847239    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"]
Oct 30 03:50:39.157876 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:39.157845    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" path="/var/lib/kubelet/pods/3626f7f5-68a5-4296-8ecb-db1479620ed9/volumes"
Oct 30 03:50:39.158351 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:39.158338    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" path="/var/lib/kubelet/pods/9527f9c0-f91a-4e27-89c1-9463688a3269/volumes"
Oct 30 03:50:39.159124 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:39.159107    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37152f2-a53d-4103-bc5a-c3aa0ca60c51" path="/var/lib/kubelet/pods/e37152f2-a53d-4103-bc5a-c3aa0ca60c51/volumes"
Oct 30 03:50:50.977119 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:50:50.977075    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:50:59.629958 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.629923    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"]
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630203    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="thanos-sidecar"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630213    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="thanos-sidecar"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630225    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="alertmanager"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630230    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="alertmanager"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630244    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="prometheus"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630249    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="prometheus"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630257    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e37152f2-a53d-4103-bc5a-c3aa0ca60c51" containerName="otc-container"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630263    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37152f2-a53d-4103-bc5a-c3aa0ca60c51" containerName="otc-container"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630270    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="init-config-reloader"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630274    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="init-config-reloader"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630282    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="config-reloader"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630287    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="config-reloader"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630293    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="config-reloader"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630298    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="config-reloader"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630308    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="init-config-reloader"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630313    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="init-config-reloader"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630320    4334 cpu_manager.go:414] "containerMap: removing container" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" containerName="prom-label-proxy"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630325    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" containerName="prom-label-proxy"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630335    4334 cpu_manager.go:414] "containerMap: removing container" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" containerName="kube-rbac-proxy"
Oct 30 03:50:59.630357 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630339    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" containerName="kube-rbac-proxy"
Oct 30 03:50:59.630960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630380    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="thanos-sidecar"
Oct 30 03:50:59.630960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630389    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="alertmanager"
Oct 30 03:50:59.630960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630397    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="prometheus"
Oct 30 03:50:59.630960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630402    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37152f2-a53d-4103-bc5a-c3aa0ca60c51" containerName="otc-container"
Oct 30 03:50:59.630960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630408    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9527f9c0-f91a-4e27-89c1-9463688a3269" containerName="config-reloader"
Oct 30 03:50:59.630960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630414    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="3626f7f5-68a5-4296-8ecb-db1479620ed9" containerName="config-reloader"
Oct 30 03:50:59.630960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630420    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" containerName="prom-label-proxy"
Oct 30 03:50:59.630960 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.630426    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="94dadde8-481e-4bc7-827a-0628180c7c20" containerName="kube-rbac-proxy"
Oct 30 03:50:59.633204 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.633188    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.653358 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.653332    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"]
Oct 30 03:50:59.710104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.710076    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.710104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.710104    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.710322 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.710123    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg2g2\" (UniqueName: \"kubernetes.io/projected/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-kube-api-access-mg2g2\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.710322 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.710241    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-config\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.710439 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.710342    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.725744 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.725720    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:50:59.725849 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.725723    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\""
Oct 30 03:50:59.725849 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.725791    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\""
Oct 30 03:50:59.725849 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.725792    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:50:59.728239 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.728222    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\""
Oct 30 03:50:59.738773 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.738754    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-ssglj\""
Oct 30 03:50:59.738845 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.738786    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\""
Oct 30 03:50:59.811575 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.811546    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.811575 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.811579    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.811776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.811603    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2g2\" (UniqueName: \"kubernetes.io/projected/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-kube-api-access-mg2g2\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.811776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.811643    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-config\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.811776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.811718    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.812371 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.812347    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.812528 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.812422    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-config\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.814220 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.814193    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.814500 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.814483    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.836911 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.836891    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2g2\" (UniqueName: \"kubernetes.io/projected/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-kube-api-access-mg2g2\") pod \"data-science-lokistack-distributor-58947679b7-6z2ll\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:50:59.904431 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.904409    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"]
Oct 30 03:50:59.908763 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.908745    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:50:59.926156 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.926133    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"]
Oct 30 03:50:59.944182 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:50:59.944161    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:51:00.001993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.001860    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\""
Oct 30 03:51:00.001993 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.001959    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\""
Oct 30 03:51:00.002146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.002030    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-shared-secret\""
Oct 30 03:51:00.015480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.014306    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.015480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.014490    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.015480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.014546    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.015480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.014598    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.015480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.014711    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-config\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.015480 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.014778    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qml4\" (UniqueName: \"kubernetes.io/projected/d7fd8285-4150-446e-8385-1f3bc8a1ca81-kube-api-access-5qml4\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.115780 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.115742    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.115975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.115792    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.115975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.115834    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.115975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.115868    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-config\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.115975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.115919    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qml4\" (UniqueName: \"kubernetes.io/projected/d7fd8285-4150-446e-8385-1f3bc8a1ca81-kube-api-access-5qml4\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.115975 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.115957    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.116683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.116657    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.116800 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.116725    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-config\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.118349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.118330    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.118799 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.118778    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.118898 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.118883    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.140683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.140659    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qml4\" (UniqueName: \"kubernetes.io/projected/d7fd8285-4150-446e-8385-1f3bc8a1ca81-kube-api-access-5qml4\") pod \"data-science-lokistack-querier-84985db556-q6j9p\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.220240 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.220171    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:00.250418 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.250364    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"]
Oct 30 03:51:00.253047 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:51:00.253022    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67d74c6_fcd5_4dc2_9e27_6b20b2ac70e7.slice/crio-4285a0cc6fd801e08040e1fa50bb68df1fdcd38758f0807f15218a1049b2a855 WatchSource:0}: Error finding container 4285a0cc6fd801e08040e1fa50bb68df1fdcd38758f0807f15218a1049b2a855: Status 404 returned error can't find the container with id 4285a0cc6fd801e08040e1fa50bb68df1fdcd38758f0807f15218a1049b2a855
Oct 30 03:51:00.255077 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.255056    4334 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:51:00.284658 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.284618    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"]
Oct 30 03:51:00.289999 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.289975    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.310471 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.310415    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"]
Oct 30 03:51:00.311005 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.310978    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-ndbrm\""
Oct 30 03:51:00.312118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.311769    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\""
Oct 30 03:51:00.312118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.311854    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 03:51:00.312118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.311905    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\""
Oct 30 03:51:00.312118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.311964    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 03:51:00.312118 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.312083    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\""
Oct 30 03:51:00.418938 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.418907    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-lokistack-gateway\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.419135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.418951    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-tls-secret\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.419135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.418976    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rq5\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-kube-api-access-x6rq5\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.419135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.419083    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-tenants\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.419135 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.419127    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.419301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.419184    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.419301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.419232    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-rbac\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.419301 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.419248    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.519693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.519612    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-tls-secret\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.519693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.519654    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rq5\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-kube-api-access-x6rq5\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.519693 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.519680    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-tenants\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.519971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.519706    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.519971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.519745    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.519971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.519817    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-rbac\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.519971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.519840    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.519971 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.519874    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-lokistack-gateway\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.520848 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.520779    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.521033 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.520813    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-lokistack-gateway\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.521033 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.521007    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-rbac\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.522760 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.522737    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.522856 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.522802    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-tls-secret\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.523053 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.523031    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-tenants\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.523211 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.523136    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.523627 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.523605    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"]
Oct 30 03:51:00.525272 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:51:00.525251    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fd8285_4150_446e_8385_1f3bc8a1ca81.slice/crio-19e6045069c2c8e87633d74b1dfa01365ccc22eabcb07076013f228fdfac5d6f WatchSource:0}: Error finding container 19e6045069c2c8e87633d74b1dfa01365ccc22eabcb07076013f228fdfac5d6f: Status 404 returned error can't find the container with id 19e6045069c2c8e87633d74b1dfa01365ccc22eabcb07076013f228fdfac5d6f
Oct 30 03:51:00.540538 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.540510    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rq5\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-kube-api-access-x6rq5\") pod \"data-science-lokistack-gateway-7c4fb6d468-g6xfd\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.606683 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.606663    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:00.833291 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.833205    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p" event={"ID":"d7fd8285-4150-446e-8385-1f3bc8a1ca81","Type":"ContainerStarted","Data":"19e6045069c2c8e87633d74b1dfa01365ccc22eabcb07076013f228fdfac5d6f"}
Oct 30 03:51:00.834233 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.834214    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll" event={"ID":"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7","Type":"ContainerStarted","Data":"4285a0cc6fd801e08040e1fa50bb68df1fdcd38758f0807f15218a1049b2a855"}
Oct 30 03:51:00.919888 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:00.919867    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"]
Oct 30 03:51:00.921939 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:51:00.921911    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef0feaa_aab6_48cd_9046_c2a5895fd928.slice/crio-fc930e744c4320f434e59dff5ad49476c9e3d7e84ad84f60ce5a782a58c5cddc WatchSource:0}: Error finding container fc930e744c4320f434e59dff5ad49476c9e3d7e84ad84f60ce5a782a58c5cddc: Status 404 returned error can't find the container with id fc930e744c4320f434e59dff5ad49476c9e3d7e84ad84f60ce5a782a58c5cddc
Oct 30 03:51:01.844879 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:01.844841    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" event={"ID":"eef0feaa-aab6-48cd-9046-c2a5895fd928","Type":"ContainerStarted","Data":"fc930e744c4320f434e59dff5ad49476c9e3d7e84ad84f60ce5a782a58c5cddc"}
Oct 30 03:51:04.151338 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.151292    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 03:51:04.156583 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.156492    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.171623 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.171558    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 03:51:04.182700 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.182669    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\""
Oct 30 03:51:04.183395 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.183376    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\""
Oct 30 03:51:04.254312 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.254268    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.254512 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.254329    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.254512 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.254361    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8lw\" (UniqueName: \"kubernetes.io/projected/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-kube-api-access-kj8lw\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.254512 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.254390    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.254512 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.254432    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.254512 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.254475    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.355536 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.355500    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.355702 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.355544    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.355702 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.355578    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8lw\" (UniqueName: \"kubernetes.io/projected/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-kube-api-access-kj8lw\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.355702 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.355609    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.355702 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.355675    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.355835 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.355718    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.356472 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.356174    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.356472 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.356257    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.360468 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.359135    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.372572 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.361272    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.373479 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.373419    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.385614 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.385595    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8lw\" (UniqueName: \"kubernetes.io/projected/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-kube-api-access-kj8lw\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:04.866574 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.866487    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p" event={"ID":"d7fd8285-4150-446e-8385-1f3bc8a1ca81","Type":"ContainerStarted","Data":"0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37"}
Oct 30 03:51:04.866574 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.866559    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:04.868087 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.868061    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll" event={"ID":"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7","Type":"ContainerStarted","Data":"9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0"}
Oct 30 03:51:04.868226 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.868164    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:51:04.869549 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.869528    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" event={"ID":"eef0feaa-aab6-48cd-9046-c2a5895fd928","Type":"ContainerStarted","Data":"c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c"}
Oct 30 03:51:04.909105 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.909055    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p" podStartSLOduration=1.867216418 podStartE2EDuration="5.909042369s" totalImagesPullingTime="4.041825951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:04.908172169 +0000 UTC m=+4143.914762422" watchObservedRunningTime="2026-10-30 03:51:04.909042369 +0000 UTC m=+4143.915632624"
Oct 30 03:51:04.948813 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:04.948766    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll" podStartSLOduration=1.634372403 podStartE2EDuration="5.948752021s" totalImagesPullingTime="4.314379618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:04.947899828 +0000 UTC m=+4143.954490082" watchObservedRunningTime="2026-10-30 03:51:04.948752021 +0000 UTC m=+4143.955342275"
Oct 30 03:51:06.882132 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:06.882090    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" event={"ID":"eef0feaa-aab6-48cd-9046-c2a5895fd928","Type":"ContainerStarted","Data":"be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c"}
Oct 30 03:51:06.882594 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:06.882324    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:06.888525 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:06.888505    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:06.924773 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:06.924725    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" podStartSLOduration=1.9576726359999999 podStartE2EDuration="6.92470838s" totalImagesPullingTime="4.967035744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:06.923723965 +0000 UTC m=+4145.930314218" watchObservedRunningTime="2026-10-30 03:51:06.92470838 +0000 UTC m=+4145.931298637"
Oct 30 03:51:07.887325 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:07.887295    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:07.895046 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:07.895022    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 03:51:07.989456 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:07.989425    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:08.019660 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:08.019627    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:08.090221 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:08.090189    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:08.090374 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:08.090344    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:08.106024 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:08.106005    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") DevicePath \"csi-0f1856b7bc24f5054bb19284af08cb2dd4b7e41a67758b256a73cda68f694ea5\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:09.273944 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:09.273912    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d4eb71bfe31f716814a6d34044216d8e22b988b2d0de07ddb5158b1040bfa373/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:09.300590 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:09.300559    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:09.573214 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:09.573130    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:09.872549 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:09.872523    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 03:51:09.874152 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:51:09.874111    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1e5e7b_9c6c_4ccd_b560_bb0606b5925e.slice/crio-a2641685bcfb37885dc145731c4221976f277f58ee8d1ec8125719c0171a3071 WatchSource:0}: Error finding container a2641685bcfb37885dc145731c4221976f277f58ee8d1ec8125719c0171a3071: Status 404 returned error can't find the container with id a2641685bcfb37885dc145731c4221976f277f58ee8d1ec8125719c0171a3071
Oct 30 03:51:09.899216 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:09.899189    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e","Type":"ContainerStarted","Data":"a2641685bcfb37885dc145731c4221976f277f58ee8d1ec8125719c0171a3071"}
Oct 30 03:51:10.905294 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:10.905256    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e","Type":"ContainerStarted","Data":"4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907"}
Oct 30 03:51:10.905783 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:10.905348    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:10.948954 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:10.948906    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=10.948893021 podStartE2EDuration="10.948893021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:10.947408466 +0000 UTC m=+4149.953998720" watchObservedRunningTime="2026-10-30 03:51:10.948893021 +0000 UTC m=+4149.955483276"
Oct 30 03:51:20.977394 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:51:20.977355    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:51:25.882592 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:25.882561    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 03:51:25.883029 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:25.882682    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 03:51:31.915436 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:51:31.915406    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 03:51:50.978232 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:51:50.978156    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:52:09.525955 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.525917    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 03:52:09.528406 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.528389    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.559355 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.559328    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\""
Oct 30 03:52:09.559521 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.559466    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\""
Oct 30 03:52:09.561201 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.561180    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 03:52:09.689321 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.689289    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.689542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.689341    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.689542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.689420    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4r6\" (UniqueName: \"kubernetes.io/projected/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-kube-api-access-9m4r6\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.689542 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.689487    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.689690 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.689535    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.689690 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.689661    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.790789 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.790718    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.790789 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.790758    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4r6\" (UniqueName: \"kubernetes.io/projected/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-kube-api-access-9m4r6\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.791057 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.790885    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.791057 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.790922    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.791057 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.790966    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.791057 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.791035    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.791685 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.791659    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.791899 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.791876    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.793345 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.793315    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.793472 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.793357    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.793472 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.793372    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:09.812606 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:09.812586    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4r6\" (UniqueName: \"kubernetes.io/projected/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-kube-api-access-9m4r6\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.362615 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.362581    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.363004 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.362658    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.393367 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.393337    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.393499 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.393366    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.463610 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.463583    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.463721 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.463643    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.463767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.463719    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.463767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.463748    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.479121 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.479096    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") DevicePath \"csi-72f9fd7edd21381a49cd8bff25b69ca508f90e731bbfff4bb8443d7734924eab\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.479202 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.479096    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") DevicePath \"csi-99e919c9bf38997c586e856e4f52645fbccb9cb6c8f8259360d168669630ccb7\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.705263 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.705231    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7230c46a45bf313912a2ed81d5eb4c8b90077d8ed7189436af8d364ee20bac80/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:18.730489 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:18.730459    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:20.977924 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:52:20.977891    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:52:25.801131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:25.801104    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/84bf2a1793914136c47db119cd10a17b4f6b9dae3231cb12f25dff5e3840823b/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:25.826652 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:25.826623    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") pod \"data-science-lokistack-ingester-1\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:26.039069 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:26.038999    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:26.348900 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:26.348875    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 03:52:26.351382 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:52:26.351354    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3f9f7a_f1c0_4c46_b8db_d92b42bcba1e.slice/crio-1e326dd7f2954d99b03cd3f107b265beca5dde46cd7eec2771f0a6fc2bf32a62 WatchSource:0}: Error finding container 1e326dd7f2954d99b03cd3f107b265beca5dde46cd7eec2771f0a6fc2bf32a62: Status 404 returned error can't find the container with id 1e326dd7f2954d99b03cd3f107b265beca5dde46cd7eec2771f0a6fc2bf32a62
Oct 30 03:52:27.264403 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:27.264367    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e","Type":"ContainerStarted","Data":"e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a"}
Oct 30 03:52:27.264819 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:27.264414    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e","Type":"ContainerStarted","Data":"1e326dd7f2954d99b03cd3f107b265beca5dde46cd7eec2771f0a6fc2bf32a62"}
Oct 30 03:52:27.264819 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:27.264509    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:52:27.308143 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:27.308097    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podStartSLOduration=21.308084372 podStartE2EDuration="21.308084372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:27.306913235 +0000 UTC m=+4226.313503489" watchObservedRunningTime="2026-10-30 03:52:27.308084372 +0000 UTC m=+4226.314674623"
Oct 30 03:52:48.276349 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:48.276314    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:52:50.978869 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:52:50.978826    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:52:58.275878 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:52:58.275841    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:53:08.274619 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:53:08.274582    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:53:18.274979 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:53:18.274909    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 03:53:20.978321 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:53:20.978286    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:53:50.978100 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:53:50.978060    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:54:20.978424 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:54:20.978388    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:54:39.209790 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.209747    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"]
Oct 30 03:54:39.222481 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.222428    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.231106 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.231082    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"]
Oct 30 03:54:39.294394 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.294363    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhxs\" (UniqueName: \"kubernetes.io/projected/9150ad6f-162a-4d94-9d60-d97db38e5fc1-kube-api-access-5bhxs\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.294599 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.294513    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otlp-tls-certs\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.294599 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.294561    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otc-internal\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.296807 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.296788    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\""
Oct 30 03:54:39.297775 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.297756    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-4t4c9\""
Oct 30 03:54:39.304253 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.304236    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-60155212\""
Oct 30 03:54:39.395756 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.395724    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otlp-tls-certs\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.395916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.395769    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otc-internal\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.395916 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.395808    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhxs\" (UniqueName: \"kubernetes.io/projected/9150ad6f-162a-4d94-9d60-d97db38e5fc1-kube-api-access-5bhxs\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.396553 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.396527    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otc-internal\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.398208 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.398178    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otlp-tls-certs\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.415146 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.415125    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhxs\" (UniqueName: \"kubernetes.io/projected/9150ad6f-162a-4d94-9d60-d97db38e5fc1-kube-api-access-5bhxs\") pod \"data-science-usage-logs-collector-7b866bc69f-xk6vh\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.538515 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.538406    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 03:54:39.832104 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.832031    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"]
Oct 30 03:54:39.836495 ip-10-0-133-197 kubenswrapper[4334]: W1030 03:54:39.836464    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9150ad6f_162a_4d94_9d60_d97db38e5fc1.slice/crio-8d0c2691800c6dd4bbdf04914d587e7bf26964fdb61c45b30d79464b72473eac WatchSource:0}: Error finding container 8d0c2691800c6dd4bbdf04914d587e7bf26964fdb61c45b30d79464b72473eac: Status 404 returned error can't find the container with id 8d0c2691800c6dd4bbdf04914d587e7bf26964fdb61c45b30d79464b72473eac
Oct 30 03:54:39.838648 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.838616    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:54:39.838767 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.838684    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 03:54:39.901360 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.901332    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh" event={"ID":"9150ad6f-162a-4d94-9d60-d97db38e5fc1","Type":"ContainerStarted","Data":"de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473"}
Oct 30 03:54:39.901492 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.901384    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh" event={"ID":"9150ad6f-162a-4d94-9d60-d97db38e5fc1","Type":"ContainerStarted","Data":"8d0c2691800c6dd4bbdf04914d587e7bf26964fdb61c45b30d79464b72473eac"}
Oct 30 03:54:39.938746 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:54:39.938693    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh" podStartSLOduration=0.938678808 podStartE2EDuration="938.678808ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:39.937290199 +0000 UTC m=+4358.943880452" watchObservedRunningTime="2026-10-30 03:54:39.938678808 +0000 UTC m=+4358.945269114"
Oct 30 03:54:50.978356 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:54:50.978280    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:55:20.977918 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:55:20.977881    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:55:50.977932 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:55:50.977895    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:56:01.900131 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:56:01.900099    4334 scope.go:122] "RemoveContainer" containerID="27b5cfe406ca87a86eb0942d42aaf702fda6943a580bf8419bd065d9c0a54743"
Oct 30 03:56:01.908687 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:56:01.908665    4334 scope.go:122] "RemoveContainer" containerID="36fc73de7fe43fde89b7d995b8b4a9009698270331ecb4b5b9550f54133f5ee6"
Oct 30 03:56:01.916776 ip-10-0-133-197 kubenswrapper[4334]: I1030 03:56:01.916753    4334 scope.go:122] "RemoveContainer" containerID="ac4c2d64eac2a3b13a30022c57fe2b986f924abf41377df1f1fbe835f597465c"
Oct 30 03:56:20.979687 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:56:20.979603    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:56:50.978208 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:56:50.978164    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:57:20.978856 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:57:20.978820    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:57:50.978779 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:57:50.978695    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:58:20.977772 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:58:20.977734    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:58:50.978395 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:58:50.978298    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:59:20.979029 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:59:20.978948    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 03:59:50.977512 ip-10-0-133-197 kubenswrapper[4334]: E1030 03:59:50.977468    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:00:20.977905 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:00:20.977864    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:00:50.978258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:00:50.978173    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:01:20.978333 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:01:20.978293    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:01:50.979421 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:01:50.979378    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:02:20.978850 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:02:20.978764    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:02:50.978064 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:02:50.978027    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:03:20.978570 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:03:20.978533    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:03:50.978512 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:03:50.978396    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:04:20.978338 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:04:20.978301    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:04:50.977728 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:04:50.977692    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:05:20.977866 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:05:20.977788    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:05:50.977970 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:05:50.977931    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:06:20.978417 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:06:20.978377    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:06:50.979567 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:06:50.979484    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:07:20.979307 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:07:20.979269    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:07:50.978552 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:07:50.978513    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:08:20.978668 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:08:20.978585    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:08:50.977885 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:08:50.977849    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:09:20.978396 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:20.978353    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:09:50.050725 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.050652    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"]
Oct 30 04:09:50.053984 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.053963    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.063779 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.063757    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"]
Oct 30 04:09:50.160557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.160526    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47556528-df59-4bdc-8664-ea2bf3f9b399-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.160727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.160581    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krq2b\" (UniqueName: \"kubernetes.io/projected/47556528-df59-4bdc-8664-ea2bf3f9b399-kube-api-access-krq2b\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.160777 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.160725    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/47556528-df59-4bdc-8664-ea2bf3f9b399-otc-internal\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.262001 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.261963    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krq2b\" (UniqueName: \"kubernetes.io/projected/47556528-df59-4bdc-8664-ea2bf3f9b399-kube-api-access-krq2b\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.262188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.262038    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/47556528-df59-4bdc-8664-ea2bf3f9b399-otc-internal\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.262188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.262084    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47556528-df59-4bdc-8664-ea2bf3f9b399-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.262774 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.262747    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/47556528-df59-4bdc-8664-ea2bf3f9b399-otc-internal\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.264591 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.264563    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47556528-df59-4bdc-8664-ea2bf3f9b399-otlp-tls-certs\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.280580 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.280555    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krq2b\" (UniqueName: \"kubernetes.io/projected/47556528-df59-4bdc-8664-ea2bf3f9b399-kube-api-access-krq2b\") pod \"data-science-usage-logs-collector-57cc577bb6-wt4kx\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.367601 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.367529    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:50.663407 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.663376    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"]
Oct 30 04:09:50.667431 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:09:50.667403    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47556528_df59_4bdc_8664_ea2bf3f9b399.slice/crio-16da071991525ede02fdcefdc7fb8655167b37ee25744e7da801b81c8d891956 WatchSource:0}: Error finding container 16da071991525ede02fdcefdc7fb8655167b37ee25744e7da801b81c8d891956: Status 404 returned error can't find the container with id 16da071991525ede02fdcefdc7fb8655167b37ee25744e7da801b81c8d891956
Oct 30 04:09:50.669503 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.669470    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:09:50.669586 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:50.669550    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:09:50.981442 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:50.981356    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:09:51.187557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.187516    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx" event={"ID":"47556528-df59-4bdc-8664-ea2bf3f9b399","Type":"ContainerStarted","Data":"104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60"}
Oct 30 04:09:51.187959 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.187573    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx" event={"ID":"47556528-df59-4bdc-8664-ea2bf3f9b399","Type":"ContainerStarted","Data":"16da071991525ede02fdcefdc7fb8655167b37ee25744e7da801b81c8d891956"}
Oct 30 04:09:51.226124 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.226062    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx" podStartSLOduration=1.226038156 podStartE2EDuration="1.226038156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:51.223555473 +0000 UTC m=+5270.230145729" watchObservedRunningTime="2026-10-30 04:09:51.226038156 +0000 UTC m=+5270.232628412"
Oct 30 04:09:51.252681 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.252526    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"]
Oct 30 04:09:51.252967 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.252847    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh" podUID="9150ad6f-162a-4d94-9d60-d97db38e5fc1" containerName="otc-container" containerID="cri-o://de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473" gracePeriod=30
Oct 30 04:09:51.512824 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.512769    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 04:09:51.676074 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.676039    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9150ad6f-162a-4d94-9d60-d97db38e5fc1-kube-api-access-5bhxs\" (UniqueName: \"kubernetes.io/projected/9150ad6f-162a-4d94-9d60-d97db38e5fc1-kube-api-access-5bhxs\") pod \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") "
Oct 30 04:09:51.676074 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.676073    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otc-internal\") pod \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") "
Oct 30 04:09:51.676277 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.676096    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otlp-tls-certs\") pod \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\" (UID: \"9150ad6f-162a-4d94-9d60-d97db38e5fc1\") "
Oct 30 04:09:51.676518 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.676488    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otc-internal" pod "9150ad6f-162a-4d94-9d60-d97db38e5fc1" (UID: "9150ad6f-162a-4d94-9d60-d97db38e5fc1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:51.678240 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.678213    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otlp-tls-certs" pod "9150ad6f-162a-4d94-9d60-d97db38e5fc1" (UID: "9150ad6f-162a-4d94-9d60-d97db38e5fc1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:51.678344 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.678283    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9150ad6f-162a-4d94-9d60-d97db38e5fc1-kube-api-access-5bhxs" pod "9150ad6f-162a-4d94-9d60-d97db38e5fc1" (UID: "9150ad6f-162a-4d94-9d60-d97db38e5fc1"). InnerVolumeSpecName "kube-api-access-5bhxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:51.776810 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.776746    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhxs\" (UniqueName: \"kubernetes.io/projected/9150ad6f-162a-4d94-9d60-d97db38e5fc1-kube-api-access-5bhxs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:51.776810 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.776776    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:51.776810 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:51.776791    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9150ad6f-162a-4d94-9d60-d97db38e5fc1-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:52.193637 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.193602    4334 generic.go:356] "Generic (PLEG): container finished" podID="9150ad6f-162a-4d94-9d60-d97db38e5fc1" containerID="de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473" exitCode=0
Oct 30 04:09:52.194071 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.193653    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"
Oct 30 04:09:52.194071 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.193686    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh" event={"ID":"9150ad6f-162a-4d94-9d60-d97db38e5fc1","Type":"ContainerDied","Data":"de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473"}
Oct 30 04:09:52.194071 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.193737    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh" event={"ID":"9150ad6f-162a-4d94-9d60-d97db38e5fc1","Type":"ContainerDied","Data":"8d0c2691800c6dd4bbdf04914d587e7bf26964fdb61c45b30d79464b72473eac"}
Oct 30 04:09:52.194071 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.193761    4334 scope.go:122] "RemoveContainer" containerID="de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473"
Oct 30 04:09:52.202619 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.202599    4334 scope.go:122] "RemoveContainer" containerID="de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473"
Oct 30 04:09:52.202895 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:52.202876    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473\": container with ID starting with de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473 not found: ID does not exist" containerID="de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473"
Oct 30 04:09:52.202948 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.202904    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473"} err="failed to get container status \"de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473\": rpc error: code = NotFound desc = could not find container \"de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473\": container with ID starting with de80184db6e544af9bba24b2fb9d9178e331d76219e11b92de694556ef91d473 not found: ID does not exist"
Oct 30 04:09:52.241065 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.241040    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"]
Oct 30 04:09:52.252556 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.252529    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-7b866bc69f-xk6vh"]
Oct 30 04:09:52.709222 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.709189    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:09:52.709746 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.709710    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" containerName="loki-index-gateway" containerID="cri-o://4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907" gracePeriod=30
Oct 30 04:09:52.802049 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.802005    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:09:52.802356 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.802309    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" containerName="loki-ingester" containerID="cri-o://e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a" gracePeriod=600
Oct 30 04:09:52.804589 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.804567    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"]
Oct 30 04:09:52.804753 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.804734    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p" podUID="d7fd8285-4150-446e-8385-1f3bc8a1ca81" containerName="loki-querier" containerID="cri-o://0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37" gracePeriod=30
Oct 30 04:09:52.820765 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.820735    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"]
Oct 30 04:09:52.821015 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.820990    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll" podUID="d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" containerName="loki-distributor" containerID="cri-o://9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0" gracePeriod=30
Oct 30 04:09:52.845834 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.845806    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"]
Oct 30 04:09:52.846087 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.846043    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="gateway" containerID="cri-o://c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c" gracePeriod=30
Oct 30 04:09:52.846217 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.846053    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="opa" containerID="cri-o://be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c" gracePeriod=30
Oct 30 04:09:52.892334 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.892306    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:09:52.962759 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:52.962675    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"]
Oct 30 04:09:53.020837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.020806    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:53.159636 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.159603    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9150ad6f-162a-4d94-9d60-d97db38e5fc1" path="/var/lib/kubelet/pods/9150ad6f-162a-4d94-9d60-d97db38e5fc1/volumes"
Oct 30 04:09:53.189135 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.189098    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-grpc\") pod \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") "
Oct 30 04:09:53.189274 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.189147    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-kube-api-access-kj8lw\" (UniqueName: \"kubernetes.io/projected/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-kube-api-access-kj8lw\") pod \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") "
Oct 30 04:09:53.189274 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.189209    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-test-loki-shared-secret\") pod \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") "
Oct 30 04:09:53.189404 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.189282    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-http\") pod \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") "
Oct 30 04:09:53.189404 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.189340    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-ca-bundle\") pod \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") "
Oct 30 04:09:53.189404 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.189380    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-config\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-config\") pod \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") "
Oct 30 04:09:53.189657 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.189490    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\" (UID: \"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e\") "
Oct 30 04:09:53.190268 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.190215    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-ca-bundle" pod "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" (UID: "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.190708 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.190683    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-config" pod "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" (UID: "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.192355 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.192327    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-test-loki-shared-secret" pod "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" (UID: "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.192485 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.192356    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-kube-api-access-kj8lw" pod "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" (UID: "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e"). InnerVolumeSpecName "kube-api-access-kj8lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:53.192544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.192479    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-http" pod "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" (UID: "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.192544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.192491    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-grpc" pod "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" (UID: "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.193651 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.193632    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b" pod "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" (UID: "fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e"). InnerVolumeSpecName "pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:09:53.201119 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.201098    4334 generic.go:356] "Generic (PLEG): container finished" podID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerID="be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c" exitCode=0
Oct 30 04:09:53.201218 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.201168    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" event={"ID":"eef0feaa-aab6-48cd-9046-c2a5895fd928","Type":"ContainerDied","Data":"be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c"}
Oct 30 04:09:53.202474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.202436    4334 generic.go:356] "Generic (PLEG): container finished" podID="fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" containerID="4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907" exitCode=0
Oct 30 04:09:53.202575 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.202539    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:53.202575 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.202543    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e","Type":"ContainerDied","Data":"4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907"}
Oct 30 04:09:53.202691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.202588    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e","Type":"ContainerDied","Data":"a2641685bcfb37885dc145731c4221976f277f58ee8d1ec8125719c0171a3071"}
Oct 30 04:09:53.202691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.202615    4334 scope.go:122] "RemoveContainer" containerID="4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907"
Oct 30 04:09:53.203763 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.203717    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx" podUID="47556528-df59-4bdc-8664-ea2bf3f9b399" containerName="otc-container" containerID="cri-o://104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60" gracePeriod=30
Oct 30 04:09:53.219817 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.219801    4334 scope.go:122] "RemoveContainer" containerID="4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907"
Oct 30 04:09:53.220102 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:53.220081    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907\": container with ID starting with 4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907 not found: ID does not exist" containerID="4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907"
Oct 30 04:09:53.220170 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.220109    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907"} err="failed to get container status \"4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907\": rpc error: code = NotFound desc = could not find container \"4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907\": container with ID starting with 4fed52f55118dced34b10f9ad0f11e636cea6964791658a4f4c56389d7152907 not found: ID does not exist"
Oct 30 04:09:53.248905 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.248875    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:09:53.254658 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.254626    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:09:53.290782 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.290748    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.290933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.290789    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.290933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.290821    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:09:53.290933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.290838    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.290933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.290855    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj8lw\" (UniqueName: \"kubernetes.io/projected/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-kube-api-access-kj8lw\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.290933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.290869    4334 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-test-loki-shared-secret\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.290933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.290884    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.304754 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.304735    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:09:53.369925 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.369902    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 04:09:53.392288 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.392262    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0f1856b7bc24f5054bb19284af08cb2dd4b7e41a67758b256a73cda68f694ea5\""
Oct 30 04:09:53.450669 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.450645    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:53.493523 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.493439    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-http\") pod \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") "
Oct 30 04:09:53.493523 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.493490    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-grpc\") pod \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") "
Oct 30 04:09:53.493741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.493568    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-ca-bundle\") pod \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") "
Oct 30 04:09:53.493741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.493677    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7fd8285-4150-446e-8385-1f3bc8a1ca81-kube-api-access-5qml4\" (UniqueName: \"kubernetes.io/projected/d7fd8285-4150-446e-8385-1f3bc8a1ca81-kube-api-access-5qml4\") pod \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") "
Oct 30 04:09:53.493846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.493740    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-config\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-config\") pod \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") "
Oct 30 04:09:53.493846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.493784    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-test-loki-shared-secret\") pod \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\" (UID: \"d7fd8285-4150-446e-8385-1f3bc8a1ca81\") "
Oct 30 04:09:53.493964 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.493915    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-ca-bundle" pod "d7fd8285-4150-446e-8385-1f3bc8a1ca81" (UID: "d7fd8285-4150-446e-8385-1f3bc8a1ca81"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.494102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.494079    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.494548 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.494520    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-config" pod "d7fd8285-4150-446e-8385-1f3bc8a1ca81" (UID: "d7fd8285-4150-446e-8385-1f3bc8a1ca81"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.495832 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.495805    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-grpc" pod "d7fd8285-4150-446e-8385-1f3bc8a1ca81" (UID: "d7fd8285-4150-446e-8385-1f3bc8a1ca81"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.495832 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.495823    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fd8285-4150-446e-8385-1f3bc8a1ca81-kube-api-access-5qml4" pod "d7fd8285-4150-446e-8385-1f3bc8a1ca81" (UID: "d7fd8285-4150-446e-8385-1f3bc8a1ca81"). InnerVolumeSpecName "kube-api-access-5qml4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:53.496140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.496114    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-http" pod "d7fd8285-4150-446e-8385-1f3bc8a1ca81" (UID: "d7fd8285-4150-446e-8385-1f3bc8a1ca81"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.496238 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.496138    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-test-loki-shared-secret" pod "d7fd8285-4150-446e-8385-1f3bc8a1ca81" (UID: "d7fd8285-4150-446e-8385-1f3bc8a1ca81"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.594847 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.594824    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47556528-df59-4bdc-8664-ea2bf3f9b399-kube-api-access-krq2b\" (UniqueName: \"kubernetes.io/projected/47556528-df59-4bdc-8664-ea2bf3f9b399-kube-api-access-krq2b\") pod \"47556528-df59-4bdc-8664-ea2bf3f9b399\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") "
Oct 30 04:09:53.594997 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.594900    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47556528-df59-4bdc-8664-ea2bf3f9b399-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47556528-df59-4bdc-8664-ea2bf3f9b399-otlp-tls-certs\") pod \"47556528-df59-4bdc-8664-ea2bf3f9b399\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") "
Oct 30 04:09:53.594997 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.594937    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47556528-df59-4bdc-8664-ea2bf3f9b399-otc-internal\" (UniqueName: \"kubernetes.io/configmap/47556528-df59-4bdc-8664-ea2bf3f9b399-otc-internal\") pod \"47556528-df59-4bdc-8664-ea2bf3f9b399\" (UID: \"47556528-df59-4bdc-8664-ea2bf3f9b399\") "
Oct 30 04:09:53.595199 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.595183    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qml4\" (UniqueName: \"kubernetes.io/projected/d7fd8285-4150-446e-8385-1f3bc8a1ca81-kube-api-access-5qml4\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.595263 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.595204    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7fd8285-4150-446e-8385-1f3bc8a1ca81-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.595263 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.595219    4334 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-test-loki-shared-secret\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.595263 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.595235    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.595263 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.595252    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d7fd8285-4150-446e-8385-1f3bc8a1ca81-data-science-lokistack-querier-grpc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.595436 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.595412    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47556528-df59-4bdc-8664-ea2bf3f9b399-otc-internal" pod "47556528-df59-4bdc-8664-ea2bf3f9b399" (UID: "47556528-df59-4bdc-8664-ea2bf3f9b399"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.596866 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.596836    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47556528-df59-4bdc-8664-ea2bf3f9b399-kube-api-access-krq2b" pod "47556528-df59-4bdc-8664-ea2bf3f9b399" (UID: "47556528-df59-4bdc-8664-ea2bf3f9b399"). InnerVolumeSpecName "kube-api-access-krq2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:53.597442 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.597425    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47556528-df59-4bdc-8664-ea2bf3f9b399-otlp-tls-certs" pod "47556528-df59-4bdc-8664-ea2bf3f9b399" (UID: "47556528-df59-4bdc-8664-ea2bf3f9b399"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.696091 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.696023    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krq2b\" (UniqueName: \"kubernetes.io/projected/47556528-df59-4bdc-8664-ea2bf3f9b399-kube-api-access-krq2b\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.696091 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.696058    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47556528-df59-4bdc-8664-ea2bf3f9b399-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.696091 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.696071    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/47556528-df59-4bdc-8664-ea2bf3f9b399-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.782768 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.782745    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:09:53.897418 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897388    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-config\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-config\") pod \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") "
Oct 30 04:09:53.897418 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897422    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-grpc\") pod \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") "
Oct 30 04:09:53.897632 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897532    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") pod \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") "
Oct 30 04:09:53.897632 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897574    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-test-loki-shared-secret\") pod \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") "
Oct 30 04:09:53.897632 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897627    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") pod \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") "
Oct 30 04:09:53.897769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897656    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-kube-api-access-9m4r6\" (UniqueName: \"kubernetes.io/projected/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-kube-api-access-9m4r6\") pod \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") "
Oct 30 04:09:53.897769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897724    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ca-bundle\") pod \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") "
Oct 30 04:09:53.897769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897761    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-http\") pod \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\" (UID: \"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e\") "
Oct 30 04:09:53.897932 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.897819    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-config" pod "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" (UID: "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.898057 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.898029    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.898478 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.898422    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ca-bundle" pod "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" (UID: "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:53.899986 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.899960    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-grpc" pod "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" (UID: "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.900215 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.900192    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-test-loki-shared-secret" pod "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" (UID: "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.900314 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.900296    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-http" pod "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" (UID: "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:53.900472 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.900436    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-kube-api-access-9m4r6" pod "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" (UID: "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e"). InnerVolumeSpecName "kube-api-access-9m4r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:53.900653 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.900626    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5" pod "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" (UID: "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e"). InnerVolumeSpecName "pvc-6acba876-4b26-4d86-82a3-9597067b448e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:09:53.900701 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.900664    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe" pod "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" (UID: "9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e"). InnerVolumeSpecName "pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:09:53.998465 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.998420    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.998465 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.998459    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.998591 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.998475    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.998591 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.998513    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:09:53.998591 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.998525    4334 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-test-loki-shared-secret\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:53.998591 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.998537    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:09:53.998591 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:53.998548    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m4r6\" (UniqueName: \"kubernetes.io/projected/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e-kube-api-access-9m4r6\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.011516 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.011497    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:09:54.022248 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.022226    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6acba876-4b26-4d86-82a3-9597067b448e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:09:54.112803 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.107901    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-6acba876-4b26-4d86-82a3-9597067b448e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0440facc99ae98de5\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-72f9fd7edd21381a49cd8bff25b69ca508f90e731bbfff4bb8443d7734924eab\""
Oct 30 04:09:54.112803 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.107942    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-fe9d2922-c444-4fae-a23f-25a28559cb0b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-010dfbf7cdb61bbbe\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-99e919c9bf38997c586e856e4f52645fbccb9cb6c8f8259360d168669630ccb7\""
Oct 30 04:09:54.127310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.127281    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 04:09:54.166034 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.166008    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 04:09:54.211254 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.211225    4334 generic.go:356] "Generic (PLEG): container finished" podID="d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" containerID="9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0" exitCode=0
Oct 30 04:09:54.211254 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.211256    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"
Oct 30 04:09:54.211759 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.211304    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll" event={"ID":"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7","Type":"ContainerDied","Data":"9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0"}
Oct 30 04:09:54.211759 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.211358    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll" event={"ID":"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7","Type":"ContainerDied","Data":"4285a0cc6fd801e08040e1fa50bb68df1fdcd38758f0807f15218a1049b2a855"}
Oct 30 04:09:54.211759 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.211389    4334 scope.go:122] "RemoveContainer" containerID="9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0"
Oct 30 04:09:54.213093 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.213062    4334 generic.go:356] "Generic (PLEG): container finished" podID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerID="c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c" exitCode=0
Oct 30 04:09:54.213210 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.213119    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"
Oct 30 04:09:54.213210 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.213140    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" event={"ID":"eef0feaa-aab6-48cd-9046-c2a5895fd928","Type":"ContainerDied","Data":"c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c"}
Oct 30 04:09:54.213210 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.213177    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd" event={"ID":"eef0feaa-aab6-48cd-9046-c2a5895fd928","Type":"ContainerDied","Data":"fc930e744c4320f434e59dff5ad49476c9e3d7e84ad84f60ce5a782a58c5cddc"}
Oct 30 04:09:54.214656 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.214607    4334 generic.go:356] "Generic (PLEG): container finished" podID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" containerID="e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a" exitCode=0
Oct 30 04:09:54.214656 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.214648    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:09:54.214833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.214651    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e","Type":"ContainerDied","Data":"e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a"}
Oct 30 04:09:54.214833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.214698    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e","Type":"ContainerDied","Data":"1e326dd7f2954d99b03cd3f107b265beca5dde46cd7eec2771f0a6fc2bf32a62"}
Oct 30 04:09:54.216339 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.216314    4334 generic.go:356] "Generic (PLEG): container finished" podID="47556528-df59-4bdc-8664-ea2bf3f9b399" containerID="104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60" exitCode=0
Oct 30 04:09:54.216477 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.216343    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx" event={"ID":"47556528-df59-4bdc-8664-ea2bf3f9b399","Type":"ContainerDied","Data":"104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60"}
Oct 30 04:09:54.216477 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.216404    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx" event={"ID":"47556528-df59-4bdc-8664-ea2bf3f9b399","Type":"ContainerDied","Data":"16da071991525ede02fdcefdc7fb8655167b37ee25744e7da801b81c8d891956"}
Oct 30 04:09:54.216477 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.216356    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"
Oct 30 04:09:54.219065 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.219039    4334 generic.go:356] "Generic (PLEG): container finished" podID="d7fd8285-4150-446e-8385-1f3bc8a1ca81" containerID="0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37" exitCode=0
Oct 30 04:09:54.219158 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.219084    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"
Oct 30 04:09:54.219202 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.219079    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p" event={"ID":"d7fd8285-4150-446e-8385-1f3bc8a1ca81","Type":"ContainerDied","Data":"0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37"}
Oct 30 04:09:54.219244 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.219199    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p" event={"ID":"d7fd8285-4150-446e-8385-1f3bc8a1ca81","Type":"ContainerDied","Data":"19e6045069c2c8e87633d74b1dfa01365ccc22eabcb07076013f228fdfac5d6f"}
Oct 30 04:09:54.224308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.224275    4334 scope.go:122] "RemoveContainer" containerID="9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0"
Oct 30 04:09:54.224840 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:54.224814    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0\": container with ID starting with 9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0 not found: ID does not exist" containerID="9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0"
Oct 30 04:09:54.224934 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.224848    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0"} err="failed to get container status \"9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0\": rpc error: code = NotFound desc = could not find container \"9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0\": container with ID starting with 9af54af3c07f77909ced45328c642b83b7e6e98db6af1d2b3b2364c13b0345b0 not found: ID does not exist"
Oct 30 04:09:54.224934 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.224873    4334 scope.go:122] "RemoveContainer" containerID="be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c"
Oct 30 04:09:54.236587 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.236279    4334 scope.go:122] "RemoveContainer" containerID="c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c"
Oct 30 04:09:54.244703 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.244686    4334 scope.go:122] "RemoveContainer" containerID="be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c"
Oct 30 04:09:54.244937 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:54.244918    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c\": container with ID starting with be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c not found: ID does not exist" containerID="be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c"
Oct 30 04:09:54.244979 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.244944    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c"} err="failed to get container status \"be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c\": rpc error: code = NotFound desc = could not find container \"be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c\": container with ID starting with be8a82a9333f83b2a84cccc27d6180ed935a765ff1d781070fbd5ed84a2fb75c not found: ID does not exist"
Oct 30 04:09:54.244979 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.244962    4334 scope.go:122] "RemoveContainer" containerID="c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c"
Oct 30 04:09:54.245193 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:54.245177    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c\": container with ID starting with c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c not found: ID does not exist" containerID="c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c"
Oct 30 04:09:54.245240 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.245198    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c"} err="failed to get container status \"c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c\": rpc error: code = NotFound desc = could not find container \"c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c\": container with ID starting with c81ab385f72627df6f3cf23c1e0d047972aaefaedb4c4592c54f81a21256895c not found: ID does not exist"
Oct 30 04:09:54.245240 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.245211    4334 scope.go:122] "RemoveContainer" containerID="e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a"
Oct 30 04:09:54.253234 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.253212    4334 scope.go:122] "RemoveContainer" containerID="e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a"
Oct 30 04:09:54.253486 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:54.253465    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a\": container with ID starting with e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a not found: ID does not exist" containerID="e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a"
Oct 30 04:09:54.253548 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.253492    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a"} err="failed to get container status \"e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a\": rpc error: code = NotFound desc = could not find container \"e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a\": container with ID starting with e60201ca7a678d581308641fad63b4f1d42fd94b0604b1ecbda27a444464c89a not found: ID does not exist"
Oct 30 04:09:54.253548 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.253505    4334 scope.go:122] "RemoveContainer" containerID="104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60"
Oct 30 04:09:54.260279 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.260261    4334 scope.go:122] "RemoveContainer" containerID="104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60"
Oct 30 04:09:54.260543 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:54.260525    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60\": container with ID starting with 104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60 not found: ID does not exist" containerID="104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60"
Oct 30 04:09:54.260602 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.260549    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60"} err="failed to get container status \"104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60\": rpc error: code = NotFound desc = could not find container \"104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60\": container with ID starting with 104cc135785b7a05b3a83e2f4d4806d598fbbb9eb28dc0715f7df3b5d4018b60 not found: ID does not exist"
Oct 30 04:09:54.260602 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.260563    4334 scope.go:122] "RemoveContainer" containerID="0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37"
Oct 30 04:09:54.267812 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.267795    4334 scope.go:122] "RemoveContainer" containerID="0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37"
Oct 30 04:09:54.268055 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:09:54.268037    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37\": container with ID starting with 0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37 not found: ID does not exist" containerID="0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37"
Oct 30 04:09:54.268100 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.268060    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37"} err="failed to get container status \"0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37\": rpc error: code = NotFound desc = could not find container \"0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37\": container with ID starting with 0e7643fa44ef5387dae86891f32d4de65141b7b290ec4d20bd580346388f7d37 not found: ID does not exist"
Oct 30 04:09:54.309364 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309319    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-tenants\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-tenants\") pod \"eef0feaa-aab6-48cd-9046-c2a5895fd928\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") "
Oct 30 04:09:54.309437 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309376    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-http\") pod \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") "
Oct 30 04:09:54.309437 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309396    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-kube-api-access-mg2g2\" (UniqueName: \"kubernetes.io/projected/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-kube-api-access-mg2g2\") pod \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") "
Oct 30 04:09:54.309437 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309428    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-config\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-config\") pod \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") "
Oct 30 04:09:54.309584 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309465    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-client-http\") pod \"eef0feaa-aab6-48cd-9046-c2a5895fd928\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") "
Oct 30 04:09:54.309584 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309500    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-ca-bundle\") pod \"eef0feaa-aab6-48cd-9046-c2a5895fd928\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") "
Oct 30 04:09:54.309584 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309537    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-rbac\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-rbac\") pod \"eef0feaa-aab6-48cd-9046-c2a5895fd928\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") "
Oct 30 04:09:54.309584 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309569    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-kube-api-access-x6rq5\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-kube-api-access-x6rq5\") pod \"eef0feaa-aab6-48cd-9046-c2a5895fd928\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") "
Oct 30 04:09:54.309786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309601    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-lokistack-gateway\") pod \"eef0feaa-aab6-48cd-9046-c2a5895fd928\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") "
Oct 30 04:09:54.309786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309640    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-grpc\") pod \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") "
Oct 30 04:09:54.309786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309676    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-ca-bundle\") pod \"eef0feaa-aab6-48cd-9046-c2a5895fd928\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") "
Oct 30 04:09:54.309786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309738    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-ca-bundle\") pod \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\" (UID: \"d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7\") "
Oct 30 04:09:54.309786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.309772    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-tls-secret\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-tls-secret\") pod \"eef0feaa-aab6-48cd-9046-c2a5895fd928\" (UID: \"eef0feaa-aab6-48cd-9046-c2a5895fd928\") "
Oct 30 04:09:54.311837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.311798    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-ca-bundle" pod "eef0feaa-aab6-48cd-9046-c2a5895fd928" (UID: "eef0feaa-aab6-48cd-9046-c2a5895fd928"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.311960 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.311855    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-config" pod "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" (UID: "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.312058 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.312034    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-ca-bundle" pod "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" (UID: "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.312534 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.312486    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-kube-api-access-mg2g2" pod "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" (UID: "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7"). InnerVolumeSpecName "kube-api-access-mg2g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:54.312534 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.312515    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-ca-bundle" pod "eef0feaa-aab6-48cd-9046-c2a5895fd928" (UID: "eef0feaa-aab6-48cd-9046-c2a5895fd928"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:54.312749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.312628    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-tls-secret" pod "eef0feaa-aab6-48cd-9046-c2a5895fd928" (UID: "eef0feaa-aab6-48cd-9046-c2a5895fd928"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:54.313023 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.313001    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"]
Oct 30 04:09:54.313422 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.313381    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-kube-api-access-x6rq5" pod "eef0feaa-aab6-48cd-9046-c2a5895fd928" (UID: "eef0feaa-aab6-48cd-9046-c2a5895fd928"). InnerVolumeSpecName "kube-api-access-x6rq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:54.313806 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.313757    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-http" pod "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" (UID: "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:54.314747 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.314721    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-grpc" pod "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" (UID: "d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:54.314886 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.314870    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-client-http" pod "eef0feaa-aab6-48cd-9046-c2a5895fd928" (UID: "eef0feaa-aab6-48cd-9046-c2a5895fd928"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:54.316308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.316286    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-rbac" pod "eef0feaa-aab6-48cd-9046-c2a5895fd928" (UID: "eef0feaa-aab6-48cd-9046-c2a5895fd928"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.316386 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.316332    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-57cc577bb6-wt4kx"]
Oct 30 04:09:54.317384 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.317361    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-lokistack-gateway" pod "eef0feaa-aab6-48cd-9046-c2a5895fd928" (UID: "eef0feaa-aab6-48cd-9046-c2a5895fd928"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:54.318219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.318200    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-tenants" pod "eef0feaa-aab6-48cd-9046-c2a5895fd928" (UID: "eef0feaa-aab6-48cd-9046-c2a5895fd928"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:54.345379 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.345360    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:09:54.350388 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.350369    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:09:54.380818 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.380794    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"]
Oct 30 04:09:54.382638 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.382621    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-q6j9p"]
Oct 30 04:09:54.410505 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410487    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6rq5\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-kube-api-access-x6rq5\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410506    4334 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-lokistack-gateway\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410518    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410529    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410539    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410550    4334 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-tls-secret\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410563    4334 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410573    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-data-science-lokistack-distributor-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410583    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg2g2\" (UniqueName: \"kubernetes.io/projected/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-kube-api-access-mg2g2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410591    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410602    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410902 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410611    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/eef0feaa-aab6-48cd-9046-c2a5895fd928-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.410902 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.410620    4334 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/eef0feaa-aab6-48cd-9046-c2a5895fd928-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:09:54.553528 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.553496    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"]
Oct 30 04:09:54.555166 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.555145    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-6z2ll"]
Oct 30 04:09:54.587047 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.586964    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"]
Oct 30 04:09:54.590434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:54.590412    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-7c4fb6d468-g6xfd"]
Oct 30 04:09:55.159574 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:55.159537    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47556528-df59-4bdc-8664-ea2bf3f9b399" path="/var/lib/kubelet/pods/47556528-df59-4bdc-8664-ea2bf3f9b399/volumes"
Oct 30 04:09:55.160012 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:55.159991    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" path="/var/lib/kubelet/pods/9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e/volumes"
Oct 30 04:09:55.160399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:55.160382    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" path="/var/lib/kubelet/pods/d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7/volumes"
Oct 30 04:09:55.161079 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:55.161057    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7fd8285-4150-446e-8385-1f3bc8a1ca81" path="/var/lib/kubelet/pods/d7fd8285-4150-446e-8385-1f3bc8a1ca81/volumes"
Oct 30 04:09:55.161475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:55.161443    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" path="/var/lib/kubelet/pods/eef0feaa-aab6-48cd-9046-c2a5895fd928/volumes"
Oct 30 04:09:55.161915 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:55.161898    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" path="/var/lib/kubelet/pods/fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e/volumes"
Oct 30 04:09:56.911834 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.911797    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"]
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912240    4334 cpu_manager.go:414] "containerMap: removing container" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="gateway"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912259    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="gateway"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912275    4334 cpu_manager.go:414] "containerMap: removing container" podUID="47556528-df59-4bdc-8664-ea2bf3f9b399" containerName="otc-container"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912285    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47556528-df59-4bdc-8664-ea2bf3f9b399" containerName="otc-container"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912301    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9150ad6f-162a-4d94-9d60-d97db38e5fc1" containerName="otc-container"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912309    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9150ad6f-162a-4d94-9d60-d97db38e5fc1" containerName="otc-container"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912323    4334 cpu_manager.go:414] "containerMap: removing container" podUID="d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" containerName="loki-distributor"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912331    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" containerName="loki-distributor"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912343    4334 cpu_manager.go:414] "containerMap: removing container" podUID="d7fd8285-4150-446e-8385-1f3bc8a1ca81" containerName="loki-querier"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912351    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7fd8285-4150-446e-8385-1f3bc8a1ca81" containerName="loki-querier"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912361    4334 cpu_manager.go:414] "containerMap: removing container" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="opa"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912369    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="opa"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912397    4334 cpu_manager.go:414] "containerMap: removing container" podUID="fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" containerName="loki-index-gateway"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912404    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" containerName="loki-index-gateway"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912418    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" containerName="loki-ingester"
Oct 30 04:09:56.912417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912425    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" containerName="loki-ingester"
Oct 30 04:09:56.913236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912518    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9150ad6f-162a-4d94-9d60-d97db38e5fc1" containerName="otc-container"
Oct 30 04:09:56.913236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912534    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67d74c6-fcd5-4dc2-9e27-6b20b2ac70e7" containerName="loki-distributor"
Oct 30 04:09:56.913236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912544    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7fd8285-4150-446e-8385-1f3bc8a1ca81" containerName="loki-querier"
Oct 30 04:09:56.913236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912552    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="opa"
Oct 30 04:09:56.913236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912564    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1e5e7b-9c6c-4ccd-b560-bb0606b5925e" containerName="loki-index-gateway"
Oct 30 04:09:56.913236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912574    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3f9f7a-f1c0-4c46-b8db-d92b42bcba1e" containerName="loki-ingester"
Oct 30 04:09:56.913236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912584    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef0feaa-aab6-48cd-9046-c2a5895fd928" containerName="gateway"
Oct 30 04:09:56.913236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.912593    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="47556528-df59-4bdc-8664-ea2bf3f9b399" containerName="otc-container"
Oct 30 04:09:56.917889 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.917862    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:56.930088 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.930059    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:56.930182 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.930104    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:56.930229 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.930173    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:56.930265 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.930239    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-config\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:56.930328 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.930315    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwm6s\" (UniqueName: \"kubernetes.io/projected/f6339eb1-05de-487e-b060-41d6a48f047e-kube-api-access-pwm6s\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:56.933938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.933879    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"]
Oct 30 04:09:56.942651 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.942628    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:09:56.942781 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.942658    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\""
Oct 30 04:09:56.942781 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.942708    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:09:56.945427 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.945406    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\""
Oct 30 04:09:56.945554 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.945499    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\""
Oct 30 04:09:56.948645 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.948613    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\""
Oct 30 04:09:56.948735 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:56.948672    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-sk4rz\""
Oct 30 04:09:57.004017 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.003990    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:09:57.007619 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.007596    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.027599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.027573    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:09:57.030909 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.030886    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwm6s\" (UniqueName: \"kubernetes.io/projected/f6339eb1-05de-487e-b060-41d6a48f047e-kube-api-access-pwm6s\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.031039 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.030937    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.031039 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.030962    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.031170 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.031123    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.031225 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.031201    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-config\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.031724 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.031695    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.031986 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.031967    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-config\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.033643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.033623    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.033833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.033815    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.037701 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.037676    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\""
Oct 30 04:09:57.037829 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.037815    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\""
Oct 30 04:09:57.037898 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.037841    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\""
Oct 30 04:09:57.049016 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.048993    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwm6s\" (UniqueName: \"kubernetes.io/projected/f6339eb1-05de-487e-b060-41d6a48f047e-kube-api-access-pwm6s\") pod \"data-science-lokistack-distributor-58947679b7-t4q6n\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.131843 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.131811    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.131843 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.131844    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.132104 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.131884    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.132104 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.132062    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.132225 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.132112    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.132282 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.132218    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qrr\" (UniqueName: \"kubernetes.io/projected/9d6e6288-6ef9-4b3b-8729-e091b83f0411-kube-api-access-58qrr\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.177376 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.177286    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:09:57.181069 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.181047    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.197764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.197717    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:09:57.232609 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.232578    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:57.232800 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.232771    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.232860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.232799    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.232860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.232831    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.232955 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.232883    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.232955 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.232912    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.233067 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233036    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.233115 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233098    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.233172 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233147    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.233213 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233199    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.233277 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233234    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.233391 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233370    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9h8w\" (UniqueName: \"kubernetes.io/projected/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-kube-api-access-s9h8w\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.233514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233480    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qrr\" (UniqueName: \"kubernetes.io/projected/9d6e6288-6ef9-4b3b-8729-e091b83f0411-kube-api-access-58qrr\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.233735 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233568    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.233860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.233611    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.235867 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.235841    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.235955 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.235865    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.235955 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.235906    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.238140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.238117    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-grpc\""
Oct 30 04:09:57.238228 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.238206    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-http\""
Oct 30 04:09:57.252237 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.252210    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qrr\" (UniqueName: \"kubernetes.io/projected/9d6e6288-6ef9-4b3b-8729-e091b83f0411-kube-api-access-58qrr\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:09:57.334180 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.334144    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.334180 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.334180    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.334425 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.334203    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.334425 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.334225    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.334425 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.334268    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9h8w\" (UniqueName: \"kubernetes.io/projected/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-kube-api-access-s9h8w\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.334425 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.334360    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.335145 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.335114    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.335275 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.335178    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.336881 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.336859    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.337000 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.336898    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.337563 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.337545    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.341652 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.341629    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:09:57.346381 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.346360    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.352867 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.352823    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9h8w\" (UniqueName: \"kubernetes.io/projected/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-kube-api-access-s9h8w\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:09:57.364211 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.364181    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:09:57.408938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.408907    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\""
Oct 30 04:09:57.409084 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.409041    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\""
Oct 30 04:09:57.493262 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.493048    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"]
Oct 30 04:09:57.497035 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.497009    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.513600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.513576    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\""
Oct 30 04:09:57.515085 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.514377    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\""
Oct 30 04:09:57.515085 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.514429    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 04:09:57.516239 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.516214    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"]
Oct 30 04:09:57.525647 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.525622    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\""
Oct 30 04:09:57.525749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.525683    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 04:09:57.536441 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.536411    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.536573 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.536496    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5x5\" (UniqueName: \"kubernetes.io/projected/6039fa47-2816-4c0d-8b12-a35618c75b80-kube-api-access-7d5x5\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.536573 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.536534    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.536707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.536584    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.536707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.536609    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.536707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.536668    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.538021 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.536928    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"]
Oct 30 04:09:57.637752 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.637721    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.637926 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.637771    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.637926 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.637857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5x5\" (UniqueName: \"kubernetes.io/projected/6039fa47-2816-4c0d-8b12-a35618c75b80-kube-api-access-7d5x5\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.637926 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.637890    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.638047 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.637934    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.638047 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.637960    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.638047 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.637989    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.638157 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.638050    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.638192 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.638160    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.638333 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.638312    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.638410 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.638354    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z84th\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-kube-api-access-z84th\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.638493 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.638420    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.638548 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.638487    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tls-secret\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.638589 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.638543    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.638794 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.638767    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.639261 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.639236    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.640913 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.640874    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.640996 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.640917    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.641216 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.641196    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.655411 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.655390    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5x5\" (UniqueName: \"kubernetes.io/projected/6039fa47-2816-4c0d-8b12-a35618c75b80-kube-api-access-7d5x5\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:09:57.739335 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.739302    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z84th\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-kube-api-access-z84th\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.739550 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.739357    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.739550 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.739389    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tls-secret\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.739550 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.739427    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.739550 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.739480    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.739550 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.739534    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.740813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.739558    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.740813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.739591    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.740813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.740363    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.740813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.740381    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.740813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.740409    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.742156 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.742135    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.742156 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.742151    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tls-secret\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.742305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.742140    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.742305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.742219    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.756711 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.756663    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z84th\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-kube-api-access-z84th\") pod \"data-science-lokistack-gateway-686554c9b7-pkvxm\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:57.813179 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:57.813149    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:58.123215 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:58.123133    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"]
Oct 30 04:09:58.127501 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:09:58.127476    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f4da9a_4b9b_4ebd_906e_dbe333a1c76d.slice/crio-be05eb210b5ef8c60308b4dedd178792f978dd60fae57567de018a0a5cb975a9 WatchSource:0}: Error finding container be05eb210b5ef8c60308b4dedd178792f978dd60fae57567de018a0a5cb975a9: Status 404 returned error can't find the container with id be05eb210b5ef8c60308b4dedd178792f978dd60fae57567de018a0a5cb975a9
Oct 30 04:09:58.248808 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:58.248777    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" event={"ID":"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d","Type":"ContainerStarted","Data":"8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542"}
Oct 30 04:09:58.248971 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:58.248836    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" event={"ID":"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d","Type":"ContainerStarted","Data":"be05eb210b5ef8c60308b4dedd178792f978dd60fae57567de018a0a5cb975a9"}
Oct 30 04:09:58.250420 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:58.250389    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n" event={"ID":"f6339eb1-05de-487e-b060-41d6a48f047e","Type":"ContainerStarted","Data":"c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f"}
Oct 30 04:09:58.250564 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:58.250441    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n" event={"ID":"f6339eb1-05de-487e-b060-41d6a48f047e","Type":"ContainerStarted","Data":"ef5e81a44eef970e245b1f77ea1fa91a0930e52cdd2f43737ad3b4944e4962ea"}
Oct 30 04:09:58.250635 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:58.250590    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:09:58.289784 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:58.289719    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n" podStartSLOduration=2.289700493 podStartE2EDuration="2.289700493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:58.287092054 +0000 UTC m=+5277.293682309" watchObservedRunningTime="2026-10-30 04:09:58.289700493 +0000 UTC m=+5277.296290748"
Oct 30 04:09:59.257050 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:59.257015    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" event={"ID":"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d","Type":"ContainerStarted","Data":"a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b"}
Oct 30 04:09:59.257522 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:59.257327    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:59.257522 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:59.257367    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:59.263691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:59.263670    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:59.266041 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:59.266021    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:09:59.320463 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:09:59.320406    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" podStartSLOduration=2.3203938060000002 podStartE2EDuration="2.320393806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:59.318890619 +0000 UTC m=+5278.325480874" watchObservedRunningTime="2026-10-30 04:09:59.320393806 +0000 UTC m=+5278.326984087"
Oct 30 04:10:01.067105 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.067073    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:01.094962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.094938    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:01.168275 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.168247    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:01.168415 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.168367    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:01.181202 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.181182    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") DevicePath \"csi-0f1856b7bc24f5054bb19284af08cb2dd4b7e41a67758b256a73cda68f694ea5\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:01.280190 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.280165    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d4eb71bfe31f716814a6d34044216d8e22b988b2d0de07ddb5158b1040bfa373/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:01.304423 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.304394    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:01.559307 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.559280    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:01.856558 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.856432    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:10:01.862355 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:10:01.862322    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6039fa47_2816_4c0d_8b12_a35618c75b80.slice/crio-65ce94b1b8153fb2de6ae4a89f71e8599af4aa1687db5717b3d0ab4b92101ec7 WatchSource:0}: Error finding container 65ce94b1b8153fb2de6ae4a89f71e8599af4aa1687db5717b3d0ab4b92101ec7: Status 404 returned error can't find the container with id 65ce94b1b8153fb2de6ae4a89f71e8599af4aa1687db5717b3d0ab4b92101ec7
Oct 30 04:10:01.873727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.873699    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:01.900050 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.900027    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:01.975126 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.975095    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:01.975300 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.975243    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:01.988536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:01.988507    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") DevicePath \"csi-3b3e3be3541e19d76719f6dde64b13a3599959b467b1595645c946e1919703bd\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:02.093203 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:02.093168    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a18a593227275cf9c0182e316398b7a2c249d11f860a234713a451ed47265e62/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:02.117928 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:02.117867    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"data-science-lokistack-compactor-0\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:02.274129 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:02.274094    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"6039fa47-2816-4c0d-8b12-a35618c75b80","Type":"ContainerStarted","Data":"d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119"}
Oct 30 04:10:02.274280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:02.274146    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"6039fa47-2816-4c0d-8b12-a35618c75b80","Type":"ContainerStarted","Data":"65ce94b1b8153fb2de6ae4a89f71e8599af4aa1687db5717b3d0ab4b92101ec7"}
Oct 30 04:10:02.274280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:02.274214    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:02.293632 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:02.293610    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:02.312005 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:02.311962    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=5.311950283 podStartE2EDuration="5.311950283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:02.310487558 +0000 UTC m=+5281.317077814" watchObservedRunningTime="2026-10-30 04:10:02.311950283 +0000 UTC m=+5281.318540536"
Oct 30 04:10:02.591166 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:02.591130    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:10:02.594590 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:10:02.594566    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aee5c8d_804c_4b1a_92f8_c2007a2273a5.slice/crio-839bf183f853c6f619ba3856783b7c16f9095945bc080b0079be56c38be5b132 WatchSource:0}: Error finding container 839bf183f853c6f619ba3856783b7c16f9095945bc080b0079be56c38be5b132: Status 404 returned error can't find the container with id 839bf183f853c6f619ba3856783b7c16f9095945bc080b0079be56c38be5b132
Oct 30 04:10:03.280061 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:03.280025    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"9aee5c8d-804c-4b1a-92f8-c2007a2273a5","Type":"ContainerStarted","Data":"6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d"}
Oct 30 04:10:03.280440 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:03.280080    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"9aee5c8d-804c-4b1a-92f8-c2007a2273a5","Type":"ContainerStarted","Data":"839bf183f853c6f619ba3856783b7c16f9095945bc080b0079be56c38be5b132"}
Oct 30 04:10:03.280440 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:03.280184    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:03.326245 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:03.326191    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podStartSLOduration=6.326179507 podStartE2EDuration="6.326179507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:03.317190884 +0000 UTC m=+5282.323781139" watchObservedRunningTime="2026-10-30 04:10:03.326179507 +0000 UTC m=+5282.332769761"
Oct 30 04:10:07.622465 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.622421    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:07.648733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.648704    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:07.723230 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.723204    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:07.723355 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.723318    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:07.736362 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.736340    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") DevicePath \"csi-ec030f5072c0cf71d4dcbdfac1934c53a33550c8acff3f94f4438df9d01b2c51\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:07.835218 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.835187    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/24cb2292f4f1a6dc2e1530c8f0fcd532bb4fab1a28a419ddab25747e5cf108bd/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:07.865993 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.865966    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:07.924690 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.924661    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:07.950930 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:07.950910    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:08.025281 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:08.025255    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:08.025387 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:08.025368    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:08.038576 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:08.038558    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") DevicePath \"csi-c197de43b8555f042e650fd16c4c300bb2407911294d305d869d86f7a2001ccb\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:08.144264 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:08.144236    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/eb1c0e88ab61acf9c34bfdc8d9f5526591662938fd3d7e5232b08417061ca4a2/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:08.168215 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:08.168190    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"data-science-lokistack-ingester-0\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:08.420165 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:08.420145    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:08.732771 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:08.732574    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:10:08.736327 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:10:08.736300    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6e6288_6ef9_4b3b_8729_e091b83f0411.slice/crio-37685a5ed2ba237d398a980cc1dc6562f851afd468c8efbc5f68f25d80db1a1c WatchSource:0}: Error finding container 37685a5ed2ba237d398a980cc1dc6562f851afd468c8efbc5f68f25d80db1a1c: Status 404 returned error can't find the container with id 37685a5ed2ba237d398a980cc1dc6562f851afd468c8efbc5f68f25d80db1a1c
Oct 30 04:10:09.309183 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:09.309146    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"9d6e6288-6ef9-4b3b-8729-e091b83f0411","Type":"ContainerStarted","Data":"ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a"}
Oct 30 04:10:09.309379 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:09.309210    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:09.309379 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:09.309232    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"9d6e6288-6ef9-4b3b-8729-e091b83f0411","Type":"ContainerStarted","Data":"37685a5ed2ba237d398a980cc1dc6562f851afd468c8efbc5f68f25d80db1a1c"}
Oct 30 04:10:09.347930 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:09.347886    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podStartSLOduration=13.347873568 podStartE2EDuration="13.347873568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:09.346277361 +0000 UTC m=+5288.352867616" watchObservedRunningTime="2026-10-30 04:10:09.347873568 +0000 UTC m=+5288.354463822"
Oct 30 04:10:19.263145 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:19.263118    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:10:20.977906 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:10:20.977871    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:10:23.286280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:23.286251    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:24.290783 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:24.290753    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:30.319220 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:30.319183    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="9d6e6288-6ef9-4b3b-8729-e091b83f0411" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:10:40.318197 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:40.318157    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="9d6e6288-6ef9-4b3b-8729-e091b83f0411" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:10:50.317923 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:10:50.317891    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:10:50.978223 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:10:50.978189    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:11:20.978667 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:11:20.978627    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:11:50.977897 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:11:50.977864    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:12:20.978828 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:20.978779    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:12:34.672807 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.672726    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"]
Oct 30 04:12:34.676343 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.676322    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:34.694673 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.694640    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"]
Oct 30 04:12:34.714751 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.714715    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8rr\" (UniqueName: \"kubernetes.io/projected/2190f723-1f8a-4b3e-864a-519d977988dd-kube-api-access-ps8rr\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:34.714895 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.714838    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:34.714947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.714893    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2190f723-1f8a-4b3e-864a-519d977988dd-otc-internal\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:34.770870 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.770826    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\""
Oct 30 04:12:34.770870 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.770871    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-cj8zf\""
Oct 30 04:12:34.771049 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.770931    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-60155212\""
Oct 30 04:12:34.815312 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.815280    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:34.815491 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.815331    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2190f723-1f8a-4b3e-864a-519d977988dd-otc-internal\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:34.815491 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.815372    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8rr\" (UniqueName: \"kubernetes.io/projected/2190f723-1f8a-4b3e-864a-519d977988dd-kube-api-access-ps8rr\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:34.815593 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:34.815494    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found
Oct 30 04:12:34.815593 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:34.815581    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs podName:2190f723-1f8a-4b3e-864a-519d977988dd nodeName:}" failed. No retries permitted until 2026-10-30 04:12:35.315560487 +0000 UTC m=+5434.322150735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs") pod "data-science-usage-logs-collector-6d8f74f69b-zndfd" (UID: "2190f723-1f8a-4b3e-864a-519d977988dd") : secret "data-science-usage-logs-collector-headless-tls" not found
Oct 30 04:12:34.815944 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.815926    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2190f723-1f8a-4b3e-864a-519d977988dd-otc-internal\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:34.834619 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:34.834595    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8rr\" (UniqueName: \"kubernetes.io/projected/2190f723-1f8a-4b3e-864a-519d977988dd-kube-api-access-ps8rr\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:35.317842 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:35.317807    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:35.320218 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:35.320192    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs\") pod \"data-science-usage-logs-collector-6d8f74f69b-zndfd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:35.592676 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:35.592595    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:35.887360 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:35.887276    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"]
Oct 30 04:12:35.892241 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:12:35.892208    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2190f723_1f8a_4b3e_864a_519d977988dd.slice/crio-a9758db3c69cb7d6afc152201c3d2ec1b4125d8d0d32655d7b318044ec126f0c WatchSource:0}: Error finding container a9758db3c69cb7d6afc152201c3d2ec1b4125d8d0d32655d7b318044ec126f0c: Status 404 returned error can't find the container with id a9758db3c69cb7d6afc152201c3d2ec1b4125d8d0d32655d7b318044ec126f0c
Oct 30 04:12:35.894492 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:35.894437    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:12:35.894574 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:35.894521    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:12:36.003081 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:36.003043    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd" event={"ID":"2190f723-1f8a-4b3e-864a-519d977988dd","Type":"ContainerStarted","Data":"e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c"}
Oct 30 04:12:36.003249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:36.003092    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd" event={"ID":"2190f723-1f8a-4b3e-864a-519d977988dd","Type":"ContainerStarted","Data":"a9758db3c69cb7d6afc152201c3d2ec1b4125d8d0d32655d7b318044ec126f0c"}
Oct 30 04:12:36.041603 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:36.041555    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd" podStartSLOduration=2.04154161 podStartE2EDuration="2.04154161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:36.041243966 +0000 UTC m=+5435.047834222" watchObservedRunningTime="2026-10-30 04:12:36.04154161 +0000 UTC m=+5435.048131921"
Oct 30 04:12:38.245132 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245095    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-ca-bundle: configmap "data-science-lokistack-ca-bundle" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245185    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:38.74516005 +0000 UTC m=+5437.751750332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-lokistack-ca-bundle" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-ca-bundle" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245238    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway: configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245307    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:38.74528763 +0000 UTC m=+5437.751877880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "rbac" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245378    4334 projected.go:291] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway-ca-bundle: configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245398    4334 projected.go:196] Error preparing data for projected volume data-science-lokistack-gateway-ca-bundle for pod redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm: configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245408    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway: configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245433    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:38.745424607 +0000 UTC m=+5437.752014852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-lokistack-gateway-ca-bundle" (UniqueName: "kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245499    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:38.745479017 +0000 UTC m=+5437.752069272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "lokistack-gateway" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245635    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway: secret "data-science-lokistack-gateway" not found
Oct 30 04:12:38.245715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.245703    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:38.745686535 +0000 UTC m=+5437.752276796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tenants" (UniqueName: "kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : secret "data-science-lokistack-gateway" not found
Oct 30 04:12:38.246348 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.246322    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-client-http: secret "data-science-lokistack-gateway-client-http" not found
Oct 30 04:12:38.246491 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.246403    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:38.746387132 +0000 UTC m=+5437.752977376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-lokistack-gateway-client-http" (UniqueName: "kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : secret "data-science-lokistack-gateway-client-http" not found
Oct 30 04:12:38.277131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.277098    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:12:38.277378 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.277331    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="6039fa47-2816-4c0d-8b12-a35618c75b80" containerName="loki-index-gateway" containerID="cri-o://d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119" gracePeriod=30
Oct 30 04:12:38.300105 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.300078    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:12:38.300344 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.300294    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="9aee5c8d-804c-4b1a-92f8-c2007a2273a5" containerName="loki-compactor" containerID="cri-o://6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d" gracePeriod=30
Oct 30 04:12:38.310873 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.310846    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:12:38.311076 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.311051    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="9d6e6288-6ef9-4b3b-8729-e091b83f0411" containerName="loki-ingester" containerID="cri-o://ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a" gracePeriod=600
Oct 30 04:12:38.325830 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.325800    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"]
Oct 30 04:12:38.326051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.326028    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n" podUID="f6339eb1-05de-487e-b060-41d6a48f047e" containerName="loki-distributor" containerID="cri-o://c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f" gracePeriod=30
Oct 30 04:12:38.352006 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.351975    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"]
Oct 30 04:12:38.489435 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.489399    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"]
Oct 30 04:12:38.489688 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.489655    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd" podUID="2190f723-1f8a-4b3e-864a-519d977988dd" containerName="otc-container" containerID="cri-o://e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c" gracePeriod=30
Oct 30 04:12:38.616278 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.616254    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:12:38.653575 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.653477    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-config\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-config\") pod \"6039fa47-2816-4c0d-8b12-a35618c75b80\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") "
Oct 30 04:12:38.653575 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.653507    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-http\") pod \"6039fa47-2816-4c0d-8b12-a35618c75b80\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") "
Oct 30 04:12:38.653740 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.653616    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") pod \"6039fa47-2816-4c0d-8b12-a35618c75b80\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") "
Oct 30 04:12:38.653740 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.653661    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-ca-bundle\") pod \"6039fa47-2816-4c0d-8b12-a35618c75b80\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") "
Oct 30 04:12:38.653740 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.653712    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-grpc\") pod \"6039fa47-2816-4c0d-8b12-a35618c75b80\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") "
Oct 30 04:12:38.653916 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.653750    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-test-loki-lifecycle-secret\") pod \"6039fa47-2816-4c0d-8b12-a35618c75b80\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") "
Oct 30 04:12:38.653916 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.653791    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6039fa47-2816-4c0d-8b12-a35618c75b80-kube-api-access-7d5x5\" (UniqueName: \"kubernetes.io/projected/6039fa47-2816-4c0d-8b12-a35618c75b80-kube-api-access-7d5x5\") pod \"6039fa47-2816-4c0d-8b12-a35618c75b80\" (UID: \"6039fa47-2816-4c0d-8b12-a35618c75b80\") "
Oct 30 04:12:38.654719 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.654687    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-ca-bundle" pod "6039fa47-2816-4c0d-8b12-a35618c75b80" (UID: "6039fa47-2816-4c0d-8b12-a35618c75b80"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:38.654893 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.654864    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-config" pod "6039fa47-2816-4c0d-8b12-a35618c75b80" (UID: "6039fa47-2816-4c0d-8b12-a35618c75b80"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:38.656768 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.656709    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-http" pod "6039fa47-2816-4c0d-8b12-a35618c75b80" (UID: "6039fa47-2816-4c0d-8b12-a35618c75b80"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:38.657739 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.657713    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-test-loki-lifecycle-secret" pod "6039fa47-2816-4c0d-8b12-a35618c75b80" (UID: "6039fa47-2816-4c0d-8b12-a35618c75b80"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:38.658146 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.658121    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b" pod "6039fa47-2816-4c0d-8b12-a35618c75b80" (UID: "6039fa47-2816-4c0d-8b12-a35618c75b80"). InnerVolumeSpecName "pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:38.659329 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.659294    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-grpc" pod "6039fa47-2816-4c0d-8b12-a35618c75b80" (UID: "6039fa47-2816-4c0d-8b12-a35618c75b80"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:38.660401 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.660380    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6039fa47-2816-4c0d-8b12-a35618c75b80-kube-api-access-7d5x5" pod "6039fa47-2816-4c0d-8b12-a35618c75b80" (UID: "6039fa47-2816-4c0d-8b12-a35618c75b80"). InnerVolumeSpecName "kube-api-access-7d5x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:38.754970 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.754897    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway: configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:38.754970 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.754921    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:12:38.754970 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.754938    4334 projected.go:291] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway-ca-bundle: configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:38.754970 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.754949    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway: secret "data-science-lokistack-gateway" not found
Oct 30 04:12:38.754970 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.754971    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:39.754951677 +0000 UTC m=+5438.761541923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "rbac" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.754975    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-client-http: secret "data-science-lokistack-gateway-client-http" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.754988    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway: configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.754942    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.754956    4334 projected.go:196] Error preparing data for projected volume data-science-lokistack-gateway-ca-bundle for pod redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm: configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.755055    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-ca-bundle: configmap "data-science-lokistack-ca-bundle" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.755013    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:39.754993156 +0000 UTC m=+5438.761583392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tenants" (UniqueName: "kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : secret "data-science-lokistack-gateway" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.755086    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.755112    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:39.755097483 +0000 UTC m=+5438.761687715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "lokistack-gateway" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.755128    4334 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-test-loki-lifecycle-secret\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.755147    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:39.755136879 +0000 UTC m=+5438.761727111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-lokistack-gateway-client-http" (UniqueName: "kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : secret "data-science-lokistack-gateway-client-http" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.755164    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5x5\" (UniqueName: \"kubernetes.io/projected/6039fa47-2816-4c0d-8b12-a35618c75b80-kube-api-access-7d5x5\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.755181    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:39.755172456 +0000 UTC m=+5438.761762694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-lokistack-gateway-ca-bundle" (UniqueName: "kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.755196    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6039fa47-2816-4c0d-8b12-a35618c75b80-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:38.755213    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:39.755205093 +0000 UTC m=+5438.761795340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-lokistack-ca-bundle" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-ca-bundle" not found
Oct 30 04:12:38.755258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.755226    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6039fa47-2816-4c0d-8b12-a35618c75b80-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.755763 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.755629    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:38.764014 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.763999    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:12:38.856389 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.856365    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs\") pod \"2190f723-1f8a-4b3e-864a-519d977988dd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") "
Oct 30 04:12:38.856553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.856430    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2190f723-1f8a-4b3e-864a-519d977988dd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2190f723-1f8a-4b3e-864a-519d977988dd-otc-internal\") pod \"2190f723-1f8a-4b3e-864a-519d977988dd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") "
Oct 30 04:12:38.856553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.856495    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2190f723-1f8a-4b3e-864a-519d977988dd-kube-api-access-ps8rr\" (UniqueName: \"kubernetes.io/projected/2190f723-1f8a-4b3e-864a-519d977988dd-kube-api-access-ps8rr\") pod \"2190f723-1f8a-4b3e-864a-519d977988dd\" (UID: \"2190f723-1f8a-4b3e-864a-519d977988dd\") "
Oct 30 04:12:38.856723 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.856700    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-20a6ad6a-ea65-4a27-8c2b-1f3cf7d3ec32\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0caa7f040b644c66b\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-0f1856b7bc24f5054bb19284af08cb2dd4b7e41a67758b256a73cda68f694ea5\""
Oct 30 04:12:38.856792 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.856770    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2190f723-1f8a-4b3e-864a-519d977988dd-otc-internal" pod "2190f723-1f8a-4b3e-864a-519d977988dd" (UID: "2190f723-1f8a-4b3e-864a-519d977988dd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:38.858544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.858524    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs" pod "2190f723-1f8a-4b3e-864a-519d977988dd" (UID: "2190f723-1f8a-4b3e-864a-519d977988dd"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:38.858544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.858537    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2190f723-1f8a-4b3e-864a-519d977988dd-kube-api-access-ps8rr" pod "2190f723-1f8a-4b3e-864a-519d977988dd" (UID: "2190f723-1f8a-4b3e-864a-519d977988dd"). InnerVolumeSpecName "kube-api-access-ps8rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:38.957616 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.957588    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2190f723-1f8a-4b3e-864a-519d977988dd-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.957616 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.957615    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2190f723-1f8a-4b3e-864a-519d977988dd-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:38.957777 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:38.957629    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8rr\" (UniqueName: \"kubernetes.io/projected/2190f723-1f8a-4b3e-864a-519d977988dd-kube-api-access-ps8rr\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.019338 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.019264    4334 generic.go:356] "Generic (PLEG): container finished" podID="6039fa47-2816-4c0d-8b12-a35618c75b80" containerID="d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119" exitCode=0
Oct 30 04:12:39.019338 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.019321    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:12:39.019576 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.019343    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"6039fa47-2816-4c0d-8b12-a35618c75b80","Type":"ContainerDied","Data":"d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119"}
Oct 30 04:12:39.019576 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.019407    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"6039fa47-2816-4c0d-8b12-a35618c75b80","Type":"ContainerDied","Data":"65ce94b1b8153fb2de6ae4a89f71e8599af4aa1687db5717b3d0ab4b92101ec7"}
Oct 30 04:12:39.019576 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.019437    4334 scope.go:122] "RemoveContainer" containerID="d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119"
Oct 30 04:12:39.020812 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.020793    4334 generic.go:356] "Generic (PLEG): container finished" podID="2190f723-1f8a-4b3e-864a-519d977988dd" containerID="e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c" exitCode=0
Oct 30 04:12:39.020920 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.020827    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"
Oct 30 04:12:39.020920 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.020868    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd" event={"ID":"2190f723-1f8a-4b3e-864a-519d977988dd","Type":"ContainerDied","Data":"e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c"}
Oct 30 04:12:39.020920 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.020918    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd" event={"ID":"2190f723-1f8a-4b3e-864a-519d977988dd","Type":"ContainerDied","Data":"a9758db3c69cb7d6afc152201c3d2ec1b4125d8d0d32655d7b318044ec126f0c"}
Oct 30 04:12:39.021102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.021051    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="gateway" containerID="cri-o://8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542" gracePeriod=30
Oct 30 04:12:39.021102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.021059    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="opa" containerID="cri-o://a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b" gracePeriod=30
Oct 30 04:12:39.038769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.038752    4334 scope.go:122] "RemoveContainer" containerID="d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119"
Oct 30 04:12:39.039057 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.039039    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119\": container with ID starting with d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119 not found: ID does not exist" containerID="d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119"
Oct 30 04:12:39.039134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.039065    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119"} err="failed to get container status \"d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119\": rpc error: code = NotFound desc = could not find container \"d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119\": container with ID starting with d0f46e91481b3ce68d4ab4454b79a3ffaaa6daf00fd5bdf3cd45dd2968975119 not found: ID does not exist"
Oct 30 04:12:39.039134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.039081    4334 scope.go:122] "RemoveContainer" containerID="e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c"
Oct 30 04:12:39.106259 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.106203    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:12:39.113307 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.113285    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:12:39.147519 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.147493    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"]
Oct 30 04:12:39.161233 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.161206    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6039fa47-2816-4c0d-8b12-a35618c75b80" path="/var/lib/kubelet/pods/6039fa47-2816-4c0d-8b12-a35618c75b80/volumes"
Oct 30 04:12:39.161955 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.161937    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-6d8f74f69b-zndfd"]
Oct 30 04:12:39.169116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.168893    4334 scope.go:122] "RemoveContainer" containerID="e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c"
Oct 30 04:12:39.169371 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.169260    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c\": container with ID starting with e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c not found: ID does not exist" containerID="e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c"
Oct 30 04:12:39.169371 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.169284    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c"} err="failed to get container status \"e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c\": rpc error: code = NotFound desc = could not find container \"e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c\": container with ID starting with e3f7271abfc96ac4f7066e76e3764f1ba6e2f156496ebb1fdf51e7febb6edb6c not found: ID does not exist"
Oct 30 04:12:39.258314 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.258266    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="opa" probeResult="failure" output="Get \"https://10.133.0.112:8083/ready\": dial tcp 10.133.0.112:8083: connect: connection refused"
Oct 30 04:12:39.263276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.263246    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:12:39.328779 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.328757    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:12:39.330824 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.330808    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:12:39.361882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.361861    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-http\") pod \"f6339eb1-05de-487e-b060-41d6a48f047e\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") "
Oct 30 04:12:39.361985 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.361890    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-http\") pod \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") "
Oct 30 04:12:39.361985 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.361924    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-config\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-config\") pod \"f6339eb1-05de-487e-b060-41d6a48f047e\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") "
Oct 30 04:12:39.362114 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362096    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") pod \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") "
Oct 30 04:12:39.362174 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362161    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-grpc\") pod \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") "
Oct 30 04:12:39.362225 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362210    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-grpc\") pod \"f6339eb1-05de-487e-b060-41d6a48f047e\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") "
Oct 30 04:12:39.362273 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362247    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-test-loki-lifecycle-secret\") pod \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") "
Oct 30 04:12:39.362327 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362278    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-config" pod "f6339eb1-05de-487e-b060-41d6a48f047e" (UID: "f6339eb1-05de-487e-b060-41d6a48f047e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.362382 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362325    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-ca-bundle\") pod \"f6339eb1-05de-487e-b060-41d6a48f047e\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") "
Oct 30 04:12:39.362382 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362362    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ca-bundle\") pod \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") "
Oct 30 04:12:39.362514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362483    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") pod \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") "
Oct 30 04:12:39.362569 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362525    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-config\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-config\") pod \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") "
Oct 30 04:12:39.362833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362809    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6e6288-6ef9-4b3b-8729-e091b83f0411-kube-api-access-58qrr\" (UniqueName: \"kubernetes.io/projected/9d6e6288-6ef9-4b3b-8729-e091b83f0411-kube-api-access-58qrr\") pod \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\" (UID: \"9d6e6288-6ef9-4b3b-8729-e091b83f0411\") "
Oct 30 04:12:39.362940 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362869    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6339eb1-05de-487e-b060-41d6a48f047e-kube-api-access-pwm6s\" (UniqueName: \"kubernetes.io/projected/f6339eb1-05de-487e-b060-41d6a48f047e-kube-api-access-pwm6s\") pod \"f6339eb1-05de-487e-b060-41d6a48f047e\" (UID: \"f6339eb1-05de-487e-b060-41d6a48f047e\") "
Oct 30 04:12:39.363010 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.362974    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-ca-bundle" pod "f6339eb1-05de-487e-b060-41d6a48f047e" (UID: "f6339eb1-05de-487e-b060-41d6a48f047e"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.363612 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.363413    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-config" pod "9d6e6288-6ef9-4b3b-8729-e091b83f0411" (UID: "9d6e6288-6ef9-4b3b-8729-e091b83f0411"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.363612 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.363526    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ca-bundle" pod "9d6e6288-6ef9-4b3b-8729-e091b83f0411" (UID: "9d6e6288-6ef9-4b3b-8729-e091b83f0411"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:39.363612 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.363601    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.363848 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.363622    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.363848 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.363636    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.365782 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.365576    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-http" pod "9d6e6288-6ef9-4b3b-8729-e091b83f0411" (UID: "9d6e6288-6ef9-4b3b-8729-e091b83f0411"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.367838 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.367409    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-grpc" pod "9d6e6288-6ef9-4b3b-8729-e091b83f0411" (UID: "9d6e6288-6ef9-4b3b-8729-e091b83f0411"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.367838 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.367481    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6339eb1-05de-487e-b060-41d6a48f047e-kube-api-access-pwm6s" pod "f6339eb1-05de-487e-b060-41d6a48f047e" (UID: "f6339eb1-05de-487e-b060-41d6a48f047e"). InnerVolumeSpecName "kube-api-access-pwm6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:39.367838 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.367720    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de" pod "9d6e6288-6ef9-4b3b-8729-e091b83f0411" (UID: "9d6e6288-6ef9-4b3b-8729-e091b83f0411"). InnerVolumeSpecName "pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:39.367838 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.367746    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-http" pod "f6339eb1-05de-487e-b060-41d6a48f047e" (UID: "f6339eb1-05de-487e-b060-41d6a48f047e"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.367838 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.367810    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6e6288-6ef9-4b3b-8729-e091b83f0411-kube-api-access-58qrr" pod "9d6e6288-6ef9-4b3b-8729-e091b83f0411" (UID: "9d6e6288-6ef9-4b3b-8729-e091b83f0411"). InnerVolumeSpecName "kube-api-access-58qrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:39.368437 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.367896    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-grpc" pod "f6339eb1-05de-487e-b060-41d6a48f047e" (UID: "f6339eb1-05de-487e-b060-41d6a48f047e"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.368437 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.368018    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364" pod "9d6e6288-6ef9-4b3b-8729-e091b83f0411" (UID: "9d6e6288-6ef9-4b3b-8729-e091b83f0411"). InnerVolumeSpecName "pvc-4575788d-4b47-476e-89a1-a05676f07b7e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:39.368437 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.368063    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-test-loki-lifecycle-secret" pod "9d6e6288-6ef9-4b3b-8729-e091b83f0411" (UID: "9d6e6288-6ef9-4b3b-8729-e091b83f0411"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:39.464772 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464746    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464781    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464794    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58qrr\" (UniqueName: \"kubernetes.io/projected/9d6e6288-6ef9-4b3b-8729-e091b83f0411-kube-api-access-58qrr\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464808    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwm6s\" (UniqueName: \"kubernetes.io/projected/f6339eb1-05de-487e-b060-41d6a48f047e-kube-api-access-pwm6s\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464821    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464831    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464845    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464854    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464867    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f6339eb1-05de-487e-b060-41d6a48f047e-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.464901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.464879    4334 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9d6e6288-6ef9-4b3b-8729-e091b83f0411-test-loki-lifecycle-secret\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:39.473672 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.473656    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4575788d-4b47-476e-89a1-a05676f07b7e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:12:39.477178 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.477165    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:12:39.565314 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.565250    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-4575788d-4b47-476e-89a1-a05676f07b7e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03ea8c1ded7ee3364\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-c197de43b8555f042e650fd16c4c300bb2407911294d305d869d86f7a2001ccb\""
Oct 30 04:12:39.565314 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:39.565273    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-0948d6ae-b65d-48e2-b2e2-6c585af9efdf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02861604c3baa97de\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-ec030f5072c0cf71d4dcbdfac1934c53a33550c8acff3f94f4438df9d01b2c51\""
Oct 30 04:12:39.767089 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767054    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-ca-bundle: configmap "data-science-lokistack-ca-bundle" not found
Oct 30 04:12:39.767089 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767084    4334 projected.go:291] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway-ca-bundle: configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767105    4334 projected.go:196] Error preparing data for projected volume data-science-lokistack-gateway-ca-bundle for pod redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm: configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767107    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-client-http: secret "data-science-lokistack-gateway-client-http" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767132    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:41.767112863 +0000 UTC m=+5440.773703096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "data-science-lokistack-ca-bundle" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-ca-bundle" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767082    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway: secret "data-science-lokistack-gateway" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767140    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway: configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767064    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway: configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767150    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:41.767139781 +0000 UTC m=+5440.773730027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "data-science-lokistack-gateway-client-http" (UniqueName: "kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : secret "data-science-lokistack-gateway-client-http" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767220    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:41.767204547 +0000 UTC m=+5440.773794780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "data-science-lokistack-gateway-ca-bundle" (UniqueName: "kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway-ca-bundle" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767235    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:41.767227923 +0000 UTC m=+5440.773818154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tenants" (UniqueName: "kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : secret "data-science-lokistack-gateway" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767252    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:41.767247344 +0000 UTC m=+5440.773837575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "rbac" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:39.767309 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:39.767266    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway podName:b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d nodeName:}" failed. No retries permitted until 2026-10-30 04:12:41.76726059 +0000 UTC m=+5440.773850823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "lokistack-gateway" (UniqueName: "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway") pod "data-science-lokistack-gateway-686554c9b7-pkvxm" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d") : configmap "data-science-lokistack-gateway" not found
Oct 30 04:12:40.027813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.027780    4334 generic.go:356] "Generic (PLEG): container finished" podID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerID="a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b" exitCode=0
Oct 30 04:12:40.028059 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.027848    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" event={"ID":"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d","Type":"ContainerDied","Data":"a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b"}
Oct 30 04:12:40.029197 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.029173    4334 generic.go:356] "Generic (PLEG): container finished" podID="f6339eb1-05de-487e-b060-41d6a48f047e" containerID="c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f" exitCode=0
Oct 30 04:12:40.029316 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.029212    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"
Oct 30 04:12:40.029316 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.029259    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n" event={"ID":"f6339eb1-05de-487e-b060-41d6a48f047e","Type":"ContainerDied","Data":"c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f"}
Oct 30 04:12:40.029316 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.029305    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n" event={"ID":"f6339eb1-05de-487e-b060-41d6a48f047e","Type":"ContainerDied","Data":"ef5e81a44eef970e245b1f77ea1fa91a0930e52cdd2f43737ad3b4944e4962ea"}
Oct 30 04:12:40.029502 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.029334    4334 scope.go:122] "RemoveContainer" containerID="c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f"
Oct 30 04:12:40.036994 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.036969    4334 generic.go:356] "Generic (PLEG): container finished" podID="9d6e6288-6ef9-4b3b-8729-e091b83f0411" containerID="ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a" exitCode=0
Oct 30 04:12:40.037102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.037012    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0"
Oct 30 04:12:40.037102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.037026    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"9d6e6288-6ef9-4b3b-8729-e091b83f0411","Type":"ContainerDied","Data":"ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a"}
Oct 30 04:12:40.037102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.037087    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"9d6e6288-6ef9-4b3b-8729-e091b83f0411","Type":"ContainerDied","Data":"37685a5ed2ba237d398a980cc1dc6562f851afd468c8efbc5f68f25d80db1a1c"}
Oct 30 04:12:40.039878 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.039861    4334 scope.go:122] "RemoveContainer" containerID="c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f"
Oct 30 04:12:40.040163 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:40.040142    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f\": container with ID starting with c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f not found: ID does not exist" containerID="c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f"
Oct 30 04:12:40.040235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.040170    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f"} err="failed to get container status \"c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f\": rpc error: code = NotFound desc = could not find container \"c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f\": container with ID starting with c811e9e999e7c26b1c314e2fa818b98a3513a9d8bd195b41d19a5cb91b7a316f not found: ID does not exist"
Oct 30 04:12:40.040235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.040187    4334 scope.go:122] "RemoveContainer" containerID="ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a"
Oct 30 04:12:40.048278 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.048261    4334 scope.go:122] "RemoveContainer" containerID="ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a"
Oct 30 04:12:40.048524 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:40.048509    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a\": container with ID starting with ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a not found: ID does not exist" containerID="ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a"
Oct 30 04:12:40.048575 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.048531    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a"} err="failed to get container status \"ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a\": rpc error: code = NotFound desc = could not find container \"ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a\": container with ID starting with ac0fe50316dd23c06adf827a2946c284d9ef5423a7edc7f568eb80ef4fd89d5a not found: ID does not exist"
Oct 30 04:12:40.075350 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.075331    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"]
Oct 30 04:12:40.078568 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.078549    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n"]
Oct 30 04:12:40.121800 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.121776    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:12:40.125230 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.125209    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"]
Oct 30 04:12:40.258259 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.258224    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-t4q6n" podUID="f6339eb1-05de-487e-b060-41d6a48f047e" containerName="loki-distributor" probeResult="failure" output="Get \"https://10.133.0.108:3101/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:12:40.259528 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.259512    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:12:40.373223 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373196    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway\") pod \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") "
Oct 30 04:12:40.373387 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373261    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http\") pod \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") "
Oct 30 04:12:40.373387 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373293    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle\") pod \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") "
Oct 30 04:12:40.373387 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373326    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants\") pod \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") "
Oct 30 04:12:40.373387 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373353    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle\") pod \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") "
Oct 30 04:12:40.373627 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373394    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac\") pod \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") "
Oct 30 04:12:40.373627 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373429    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tls-secret\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tls-secret\") pod \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") "
Oct 30 04:12:40.373627 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373478    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-kube-api-access-z84th\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-kube-api-access-z84th\") pod \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\" (UID: \"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d\") "
Oct 30 04:12:40.373857 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.373814    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle" pod "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:40.375609 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.375581    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http" pod "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:40.375906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.375877    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle" pod "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:40.376167 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.376149    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-kube-api-access-z84th" pod "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d"). InnerVolumeSpecName "kube-api-access-z84th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:40.376298 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.376277    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tls-secret" pod "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:40.378551 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.378531    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac" pod "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:40.379216 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.379196    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway" pod "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:40.379907 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.379889    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants" pod "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" (UID: "b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:40.474965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.474938    4334 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-lokistack-gateway\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:40.474965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.474963    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:40.475124 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.474974    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:40.475124 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.474983    4334 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tenants\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:40.475124 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.474993    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:40.475124 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.475002    4334 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-rbac\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:40.475124 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.475010    4334 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-tls-secret\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:40.475124 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:40.475018    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z84th\" (UniqueName: \"kubernetes.io/projected/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d-kube-api-access-z84th\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:41.044952 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.044918    4334 generic.go:356] "Generic (PLEG): container finished" podID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerID="8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542" exitCode=0
Oct 30 04:12:41.045140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.044987    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"
Oct 30 04:12:41.045140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.044988    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" event={"ID":"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d","Type":"ContainerDied","Data":"8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542"}
Oct 30 04:12:41.045140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.045133    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm" event={"ID":"b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d","Type":"ContainerDied","Data":"be05eb210b5ef8c60308b4dedd178792f978dd60fae57567de018a0a5cb975a9"}
Oct 30 04:12:41.045275 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.045158    4334 scope.go:122] "RemoveContainer" containerID="a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b"
Oct 30 04:12:41.053422 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.053409    4334 scope.go:122] "RemoveContainer" containerID="8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542"
Oct 30 04:12:41.061183 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.061169    4334 scope.go:122] "RemoveContainer" containerID="a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b"
Oct 30 04:12:41.061411 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:41.061396    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b\": container with ID starting with a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b not found: ID does not exist" containerID="a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b"
Oct 30 04:12:41.061470 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.061419    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b"} err="failed to get container status \"a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b\": rpc error: code = NotFound desc = could not find container \"a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b\": container with ID starting with a6fb906188f4c51514b7085ff9f96af808cff66fc1ebec879bebb3e8dc0f5c2b not found: ID does not exist"
Oct 30 04:12:41.061470 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.061434    4334 scope.go:122] "RemoveContainer" containerID="8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542"
Oct 30 04:12:41.061663 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:41.061646    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542\": container with ID starting with 8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542 not found: ID does not exist" containerID="8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542"
Oct 30 04:12:41.061706 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.061668    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542"} err="failed to get container status \"8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542\": rpc error: code = NotFound desc = could not find container \"8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542\": container with ID starting with 8cb29d36e5b0c8d566f6ba502b9e5abdc3d9bea2454c0c3b12974061f9a04542 not found: ID does not exist"
Oct 30 04:12:41.103213 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.103192    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"]
Oct 30 04:12:41.114337 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.114316    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-686554c9b7-pkvxm"]
Oct 30 04:12:41.158374 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.158354    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2190f723-1f8a-4b3e-864a-519d977988dd" path="/var/lib/kubelet/pods/2190f723-1f8a-4b3e-864a-519d977988dd/volumes"
Oct 30 04:12:41.158761 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.158747    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6e6288-6ef9-4b3b-8729-e091b83f0411" path="/var/lib/kubelet/pods/9d6e6288-6ef9-4b3b-8729-e091b83f0411/volumes"
Oct 30 04:12:41.159076 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.159065    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" path="/var/lib/kubelet/pods/b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d/volumes"
Oct 30 04:12:41.159733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:41.159718    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6339eb1-05de-487e-b060-41d6a48f047e" path="/var/lib/kubelet/pods/f6339eb1-05de-487e-b060-41d6a48f047e/volumes"
Oct 30 04:12:44.288896 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:44.288812    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="9aee5c8d-804c-4b1a-92f8-c2007a2273a5" containerName="loki-compactor" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:12:50.977929 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:50.977883    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:12:54.289214 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:54.289170    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="9aee5c8d-804c-4b1a-92f8-c2007a2273a5" containerName="loki-compactor" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:12:59.047707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.047682    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:12:59.107638 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.107578    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") pod \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") "
Oct 30 04:12:59.107638 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.107620    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-test-loki-lifecycle-secret\") pod \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") "
Oct 30 04:12:59.107837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.107640    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-ca-bundle\") pod \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") "
Oct 30 04:12:59.107837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.107660    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-kube-api-access-s9h8w\" (UniqueName: \"kubernetes.io/projected/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-kube-api-access-s9h8w\") pod \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") "
Oct 30 04:12:59.107837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.107701    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-config\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-config\") pod \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") "
Oct 30 04:12:59.107837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.107721    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-http\") pod \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") "
Oct 30 04:12:59.107837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.107765    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-grpc\") pod \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\" (UID: \"9aee5c8d-804c-4b1a-92f8-c2007a2273a5\") "
Oct 30 04:12:59.108182 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.108041    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-ca-bundle" pod "9aee5c8d-804c-4b1a-92f8-c2007a2273a5" (UID: "9aee5c8d-804c-4b1a-92f8-c2007a2273a5"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:59.108335 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.108198    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-config" pod "9aee5c8d-804c-4b1a-92f8-c2007a2273a5" (UID: "9aee5c8d-804c-4b1a-92f8-c2007a2273a5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:59.109998 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.109970    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-test-loki-lifecycle-secret" pod "9aee5c8d-804c-4b1a-92f8-c2007a2273a5" (UID: "9aee5c8d-804c-4b1a-92f8-c2007a2273a5"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:59.110114 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.109992    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-grpc" pod "9aee5c8d-804c-4b1a-92f8-c2007a2273a5" (UID: "9aee5c8d-804c-4b1a-92f8-c2007a2273a5"). InnerVolumeSpecName "data-science-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:59.110401 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.110381    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f" pod "9aee5c8d-804c-4b1a-92f8-c2007a2273a5" (UID: "9aee5c8d-804c-4b1a-92f8-c2007a2273a5"). InnerVolumeSpecName "pvc-35cf2c66-f433-429c-932a-bb5f878f69a1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:59.110560 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.110436    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-kube-api-access-s9h8w" pod "9aee5c8d-804c-4b1a-92f8-c2007a2273a5" (UID: "9aee5c8d-804c-4b1a-92f8-c2007a2273a5"). InnerVolumeSpecName "kube-api-access-s9h8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:59.110560 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.110495    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-http" pod "9aee5c8d-804c-4b1a-92f8-c2007a2273a5" (UID: "9aee5c8d-804c-4b1a-92f8-c2007a2273a5"). InnerVolumeSpecName "data-science-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:59.129530 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.129503    4334 generic.go:356] "Generic (PLEG): container finished" podID="9aee5c8d-804c-4b1a-92f8-c2007a2273a5" containerID="6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d" exitCode=0
Oct 30 04:12:59.129625 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.129548    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:12:59.129625 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.129565    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"9aee5c8d-804c-4b1a-92f8-c2007a2273a5","Type":"ContainerDied","Data":"6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d"}
Oct 30 04:12:59.129625 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.129615    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"9aee5c8d-804c-4b1a-92f8-c2007a2273a5","Type":"ContainerDied","Data":"839bf183f853c6f619ba3856783b7c16f9095945bc080b0079be56c38be5b132"}
Oct 30 04:12:59.129739 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.129627    4334 scope.go:122] "RemoveContainer" containerID="6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d"
Oct 30 04:12:59.138261 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.138246    4334 scope.go:122] "RemoveContainer" containerID="6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d"
Oct 30 04:12:59.138555 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:12:59.138523    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d\": container with ID starting with 6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d not found: ID does not exist" containerID="6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d"
Oct 30 04:12:59.138639 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.138561    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d"} err="failed to get container status \"6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d\": rpc error: code = NotFound desc = could not find container \"6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d\": container with ID starting with 6b61349704944afe9665ad93db203d0c44fad9a7bf4366221aad2ec5fd0db55d not found: ID does not exist"
Oct 30 04:12:59.177580 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.177561    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:12:59.182958 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.182933    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:12:59.208919 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.208899    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-http\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.208919 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.208918    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-compactor-grpc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.209036 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.208942    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:12:59.209036 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.208953    4334 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-test-loki-lifecycle-secret\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.209036 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.208963    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-data-science-lokistack-ca-bundle\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.209036 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.208972    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9h8w\" (UniqueName: \"kubernetes.io/projected/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-kube-api-access-s9h8w\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.209036 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.208980    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9aee5c8d-804c-4b1a-92f8-c2007a2273a5-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.216913 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.216900    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-35cf2c66-f433-429c-932a-bb5f878f69a1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:12:59.309702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:12:59.309679    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-35cf2c66-f433-429c-932a-bb5f878f69a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0659ca0214d1b563f\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-3b3e3be3541e19d76719f6dde64b13a3599959b467b1595645c946e1919703bd\""
Oct 30 04:13:01.158482 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:01.158434    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aee5c8d-804c-4b1a-92f8-c2007a2273a5" path="/var/lib/kubelet/pods/9aee5c8d-804c-4b1a-92f8-c2007a2273a5/volumes"
Oct 30 04:13:10.401968 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.401938    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"]
Oct 30 04:13:10.402372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402288    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="gateway"
Oct 30 04:13:10.402372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402304    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="gateway"
Oct 30 04:13:10.402372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402319    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="opa"
Oct 30 04:13:10.402372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402327    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="opa"
Oct 30 04:13:10.402372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402342    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6039fa47-2816-4c0d-8b12-a35618c75b80" containerName="loki-index-gateway"
Oct 30 04:13:10.402372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402350    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6039fa47-2816-4c0d-8b12-a35618c75b80" containerName="loki-index-gateway"
Oct 30 04:13:10.402372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402362    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f6339eb1-05de-487e-b060-41d6a48f047e" containerName="loki-distributor"
Oct 30 04:13:10.402372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402368    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6339eb1-05de-487e-b060-41d6a48f047e" containerName="loki-distributor"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402380    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9aee5c8d-804c-4b1a-92f8-c2007a2273a5" containerName="loki-compactor"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402388    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aee5c8d-804c-4b1a-92f8-c2007a2273a5" containerName="loki-compactor"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402401    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9d6e6288-6ef9-4b3b-8729-e091b83f0411" containerName="loki-ingester"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402406    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6e6288-6ef9-4b3b-8729-e091b83f0411" containerName="loki-ingester"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402415    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2190f723-1f8a-4b3e-864a-519d977988dd" containerName="otc-container"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402420    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2190f723-1f8a-4b3e-864a-519d977988dd" containerName="otc-container"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402499    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="gateway"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402506    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f4da9a-4b9b-4ebd-906e-dbe333a1c76d" containerName="opa"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402512    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6039fa47-2816-4c0d-8b12-a35618c75b80" containerName="loki-index-gateway"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402520    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6339eb1-05de-487e-b060-41d6a48f047e" containerName="loki-distributor"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402525    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aee5c8d-804c-4b1a-92f8-c2007a2273a5" containerName="loki-compactor"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402532    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6e6288-6ef9-4b3b-8729-e091b83f0411" containerName="loki-ingester"
Oct 30 04:13:10.402718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.402541    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2190f723-1f8a-4b3e-864a-519d977988dd" containerName="otc-container"
Oct 30 04:13:10.407297 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.407277    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.434001 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.433977    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"]
Oct 30 04:13:10.445333 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.445309    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"]
Oct 30 04:13:10.448679 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.448661    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.465180 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.465146    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"]
Oct 30 04:13:10.489025 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.488993    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489267    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489284    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489291    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489327    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489339    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489161    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489336    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489156    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:13:10.489366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489308    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 04:13:10.489819 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489379    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/681c1cde-4758-4c4b-8d76-3105831c2f21-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.489819 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489406    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-rf86v\""
Oct 30 04:13:10.489819 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489466    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5z6\" (UniqueName: \"kubernetes.io/projected/681c1cde-4758-4c4b-8d76-3105831c2f21-kube-api-access-4w5z6\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.489819 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489515    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-j87r6\""
Oct 30 04:13:10.489819 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.489526    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.590631 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.590602    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5z6\" (UniqueName: \"kubernetes.io/projected/681c1cde-4758-4c4b-8d76-3105831c2f21-kube-api-access-4w5z6\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.590631 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.590634    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.590836 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.590663    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.590836 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.590680    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.590836 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.590799    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.590962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.590857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/681c1cde-4758-4c4b-8d76-3105831c2f21-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.590962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.590920    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.590962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.590949    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f267\" (UniqueName: \"kubernetes.io/projected/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-kube-api-access-2f267\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.591341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.591323    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.591481 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.591440    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.591522 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.591440    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.593327 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.593310    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/681c1cde-4758-4c4b-8d76-3105831c2f21-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.608635 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.608600    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5z6\" (UniqueName: \"kubernetes.io/projected/681c1cde-4758-4c4b-8d76-3105831c2f21-kube-api-access-4w5z6\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-vp28r\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.692218 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.692184    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.692371 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.692228    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.692371 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.692249    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f267\" (UniqueName: \"kubernetes.io/projected/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-kube-api-access-2f267\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.692988 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.692958    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.694828 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.694804    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.710301 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.710272    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f267\" (UniqueName: \"kubernetes.io/projected/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-kube-api-access-2f267\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hvspk\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:10.722245 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.722217    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:13:10.763889 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:10.763853    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:13:11.016862 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.016786    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"]
Oct 30 04:13:11.021200 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:13:11.021168    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681c1cde_4758_4c4b_8d76_3105831c2f21.slice/crio-df7d610e9ecf8568ca7cbfaf2903b2706cb0c865b945d58c9a56426115196565 WatchSource:0}: Error finding container df7d610e9ecf8568ca7cbfaf2903b2706cb0c865b945d58c9a56426115196565: Status 404 returned error can't find the container with id df7d610e9ecf8568ca7cbfaf2903b2706cb0c865b945d58c9a56426115196565
Oct 30 04:13:11.067112 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.067080    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"]
Oct 30 04:13:11.071234 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:13:11.071209    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3b9a83_b96c_439c_9aaa_34895adb3c7a.slice/crio-f4e64c9d908b26e87c9736c4cd5871bea72a94e8fcf2280e4c4b678ea6b06f5e WatchSource:0}: Error finding container f4e64c9d908b26e87c9736c4cd5871bea72a94e8fcf2280e4c4b678ea6b06f5e: Status 404 returned error can't find the container with id f4e64c9d908b26e87c9736c4cd5871bea72a94e8fcf2280e4c4b678ea6b06f5e
Oct 30 04:13:11.187254 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.187230    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" event={"ID":"681c1cde-4758-4c4b-8d76-3105831c2f21","Type":"ContainerStarted","Data":"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7"}
Oct 30 04:13:11.187419 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.187286    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" event={"ID":"681c1cde-4758-4c4b-8d76-3105831c2f21","Type":"ContainerStarted","Data":"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0"}
Oct 30 04:13:11.187419 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.187321    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" event={"ID":"681c1cde-4758-4c4b-8d76-3105831c2f21","Type":"ContainerStarted","Data":"df7d610e9ecf8568ca7cbfaf2903b2706cb0c865b945d58c9a56426115196565"}
Oct 30 04:13:11.188817 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.188792    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" event={"ID":"4d3b9a83-b96c-439c-9aaa-34895adb3c7a","Type":"ContainerStarted","Data":"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211"}
Oct 30 04:13:11.188899 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.188854    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" event={"ID":"4d3b9a83-b96c-439c-9aaa-34895adb3c7a","Type":"ContainerStarted","Data":"f4e64c9d908b26e87c9736c4cd5871bea72a94e8fcf2280e4c4b678ea6b06f5e"}
Oct 30 04:13:11.224478 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.224422    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" podStartSLOduration=1.224409186 podStartE2EDuration="1.224409186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:11.223710478 +0000 UTC m=+5470.230300732" watchObservedRunningTime="2026-10-30 04:13:11.224409186 +0000 UTC m=+5470.230999440"
Oct 30 04:13:11.909901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.909872    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:11.913900 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.913875    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:11.932580 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.932550    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:11.959022 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.958991    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:13:11.998489 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.998435    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:13:11.998677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.998536    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-txvtm\""
Oct 30 04:13:11.998677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.998548    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:13:11.998677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.998579    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:13:11.998677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:11.998626    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:13:12.004008 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.003983    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.004142 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.004016    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.004142 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.004041    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.004142 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.004058    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.004142 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.004123    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn9x7\" (UniqueName: \"kubernetes.io/projected/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-kube-api-access-kn9x7\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.004280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.004199    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.105104 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.105068    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.105104 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.105103    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.105347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.105126    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.105347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.105146    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.105347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.105170    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn9x7\" (UniqueName: \"kubernetes.io/projected/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-kube-api-access-kn9x7\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.105347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.105205    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.105922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.105894    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.107861 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.107837    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.107978 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.107837    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.107978 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.107901    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.108094 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.108039    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.122899 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.122878    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn9x7\" (UniqueName: \"kubernetes.io/projected/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-kube-api-access-kn9x7\") pod \"data-science-collector-collector-1\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.195343 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.195274    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" event={"ID":"4d3b9a83-b96c-439c-9aaa-34895adb3c7a","Type":"ContainerStarted","Data":"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f"}
Oct 30 04:13:12.229904 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.229878    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:12.233888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.233842    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" podStartSLOduration=2.233827166 podStartE2EDuration="2.233827166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:12.233489476 +0000 UTC m=+5471.240079731" watchObservedRunningTime="2026-10-30 04:13:12.233827166 +0000 UTC m=+5471.240417421"
Oct 30 04:13:12.535468 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.535363    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:12.539535 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:13:12.539506    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a37aa83_e184_4b1c_bc14_09c9088a4d3e.slice/crio-69aaa20e62442c87b363d4cac7e0936c466146c2ba66e89032e30aed17d06079 WatchSource:0}: Error finding container 69aaa20e62442c87b363d4cac7e0936c466146c2ba66e89032e30aed17d06079: Status 404 returned error can't find the container with id 69aaa20e62442c87b363d4cac7e0936c466146c2ba66e89032e30aed17d06079
Oct 30 04:13:12.541780 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.541745    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:13:12.541872 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:12.541809    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:13:13.201207 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:13.201166    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0a37aa83-e184-4b1c-bc14-09c9088a4d3e","Type":"ContainerStarted","Data":"89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7"}
Oct 30 04:13:13.201596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:13.201225    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0a37aa83-e184-4b1c-bc14-09c9088a4d3e","Type":"ContainerStarted","Data":"69aaa20e62442c87b363d4cac7e0936c466146c2ba66e89032e30aed17d06079"}
Oct 30 04:13:13.238640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:13.238595    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.238581674 podStartE2EDuration="2.238581674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:13.23667788 +0000 UTC m=+5472.243268133" watchObservedRunningTime="2026-10-30 04:13:13.238581674 +0000 UTC m=+5472.245171985"
Oct 30 04:13:15.203293 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.203263    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:13:15.207718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.207696    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.235785 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.235759    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:13:15.282379 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.282351    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:13:15.282379 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.282378    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:13:15.282679 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.282501    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:13:15.282679 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.282507    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:13:15.282679 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.282668    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-8ncjk\""
Oct 30 04:13:15.335886 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.335859    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv5pc\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-kube-api-access-lv5pc\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.336036 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.335900    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.336036 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.335930    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.336109 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.336055    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.336141 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.336110    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.336188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.336172    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.336224 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.336203    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.416956 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.416925    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:13:15.420910 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.420885    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.436769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.436745    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5pc\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-kube-api-access-lv5pc\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.436882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.436783    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.436882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.436804    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.436882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.436826    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.436882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.436866    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.437026 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.436901    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.437026 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.436932    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.437263 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.437233    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.439662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.439618    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.439916 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.439892    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.440105 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.440080    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.440185 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.440167    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.442089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.441620    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:13:15.442089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.442051    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.454262 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.454212    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv5pc\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-kube-api-access-lv5pc\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.517998 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.517973    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:13:15.518209 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518184    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:13:15.518341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518207    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:13:15.518341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518213    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-fw4nx\""
Oct 30 04:13:15.518341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518241    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:13:15.518341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518253    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:13:15.518341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518270    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:13:15.518341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518295    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:13:15.518341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518310    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:13:15.518619 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.518360    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:13:15.527565 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.527548    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:15.537749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.537730    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.537813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.537757    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.537813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.537774    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.537813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.537789    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.537923 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.537859    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8j7\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-kube-api-access-kx8j7\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.538056 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.537936    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.538056 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.538007    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.538056 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.538045    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.538154 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.538096    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.538191 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.538165    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.538235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.538217    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.538282 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.538251    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639228 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639199    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639231    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8j7\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-kube-api-access-kx8j7\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639252    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639277    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639302    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639351    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639370    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639780 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639406    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.639780 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.639460    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.640076 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.640014    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.640741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.640238    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.640741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.640276    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.640741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.640305    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.640741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.640329    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.640741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.640371    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.642497 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.642406    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.642618 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.642503    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.642974 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.642951    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.643211 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.643186    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.643470 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.643432    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.643562 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.643472    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.643732 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.643716    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.643854 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.643834    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.658682 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.658663    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8j7\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-kube-api-access-kx8j7\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:15.838085 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:15.838007    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:13:15.842033 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:13:15.842006    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23efc980_dc09_4412_b7cc_439358662567.slice/crio-4794e404305d7e28921c9dfdee927d601c386e9d5d4e3afd7b6c899c11c4c1bc WatchSource:0}: Error finding container 4794e404305d7e28921c9dfdee927d601c386e9d5d4e3afd7b6c899c11c4c1bc: Status 404 returned error can't find the container with id 4794e404305d7e28921c9dfdee927d601c386e9d5d4e3afd7b6c899c11c4c1bc
Oct 30 04:13:16.216343 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:16.216312    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"23efc980-dc09-4412-b7cc-439358662567","Type":"ContainerStarted","Data":"3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d"}
Oct 30 04:13:16.216716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:16.216374    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"23efc980-dc09-4412-b7cc-439358662567","Type":"ContainerStarted","Data":"4794e404305d7e28921c9dfdee927d601c386e9d5d4e3afd7b6c899c11c4c1bc"}
Oct 30 04:13:20.235929 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:20.235896    4334 generic.go:356] "Generic (PLEG): container finished" podID="23efc980-dc09-4412-b7cc-439358662567" containerID="3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d" exitCode=0
Oct 30 04:13:20.236309 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:20.235950    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"23efc980-dc09-4412-b7cc-439358662567","Type":"ContainerDied","Data":"3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d"}
Oct 30 04:13:20.977220 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:13:20.977125    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:13:21.242709 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:21.242629    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"23efc980-dc09-4412-b7cc-439358662567","Type":"ContainerStarted","Data":"174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086"}
Oct 30 04:13:21.242709 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:21.242683    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"23efc980-dc09-4412-b7cc-439358662567","Type":"ContainerStarted","Data":"7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5"}
Oct 30 04:13:21.243154 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:21.242875    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:21.305947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:21.305904    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.92190709 podStartE2EDuration="6.305891556s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.383984466s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:21.294387223 +0000 UTC m=+5480.300977488" watchObservedRunningTime="2026-10-30 04:13:21.305891556 +0000 UTC m=+5480.312481810"
Oct 30 04:13:25.525797 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:25.525766    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:25.551426 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:25.551399    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:25.626552 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:25.626524    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:25.626663 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:25.626630    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:25.639551 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:25.639529    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:25.740521 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:25.740493    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:25.766967 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:25.766943    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:25.941652 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:25.941628    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:26.282895 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:26.282816    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:13:26.288766 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:13:26.288739    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fe0ee4_679d_4a13_82a4_a01a814bc3f6.slice/crio-5fde79b4fbf6b125ca2f8b1448573bab0edca7ab2b4bc5028fc9870f5bcfd829 WatchSource:0}: Error finding container 5fde79b4fbf6b125ca2f8b1448573bab0edca7ab2b4bc5028fc9870f5bcfd829: Status 404 returned error can't find the container with id 5fde79b4fbf6b125ca2f8b1448573bab0edca7ab2b4bc5028fc9870f5bcfd829
Oct 30 04:13:27.248946 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:27.248917    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:13:27.274546 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:27.274508    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerStarted","Data":"228b1199df726ad3dcfe8c087aff71a022d7a721ede33b8cf4b6bde340f00a34"}
Oct 30 04:13:27.274677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:27.274573    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerStarted","Data":"5fde79b4fbf6b125ca2f8b1448573bab0edca7ab2b4bc5028fc9870f5bcfd829"}
Oct 30 04:13:30.289309 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:30.289274    4334 generic.go:356] "Generic (PLEG): container finished" podID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerID="228b1199df726ad3dcfe8c087aff71a022d7a721ede33b8cf4b6bde340f00a34" exitCode=0
Oct 30 04:13:30.289763 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:30.289351    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerDied","Data":"228b1199df726ad3dcfe8c087aff71a022d7a721ede33b8cf4b6bde340f00a34"}
Oct 30 04:13:31.296004 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:31.295965    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerStarted","Data":"33866d725fe7da350960c12cc516e1f8a9470412726525eca9df194ff90fe8d0"}
Oct 30 04:13:31.296400 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:31.296035    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerStarted","Data":"94e3232fe7887b1ab1c8cb3d1d509e5e3ad5dde4aa0a4f3c95cd83989b2e97ec"}
Oct 30 04:13:31.296400 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:31.296063    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerStarted","Data":"76345f3f31b0d41a7830c03e6e9a3e1d4702009940a4d253a1102e57fd4815d3"}
Oct 30 04:13:31.363271 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:31.363225    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=12.978025891 podStartE2EDuration="16.363211986s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385186095s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:31.350718039 +0000 UTC m=+5490.357308292" watchObservedRunningTime="2026-10-30 04:13:31.363211986 +0000 UTC m=+5490.369802274"
Oct 30 04:13:35.942539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:35.942506    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:40.942476 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:40.942424    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:40.947426 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:40.947406    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:41.356717 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:41.356652    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:49.955348 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:49.955316    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:13:49.955897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:49.955673    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="prometheus" containerID="cri-o://76345f3f31b0d41a7830c03e6e9a3e1d4702009940a4d253a1102e57fd4815d3" gracePeriod=600
Oct 30 04:13:49.956039 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:49.956013    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="config-reloader" containerID="cri-o://94e3232fe7887b1ab1c8cb3d1d509e5e3ad5dde4aa0a4f3c95cd83989b2e97ec" gracePeriod=600
Oct 30 04:13:49.956127 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:49.956021    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="thanos-sidecar" containerID="cri-o://33866d725fe7da350960c12cc516e1f8a9470412726525eca9df194ff90fe8d0" gracePeriod=600
Oct 30 04:13:50.289395 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.289328    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:50.289596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.289567    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0a37aa83-e184-4b1c-bc14-09c9088a4d3e" containerName="otc-container" containerID="cri-o://89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7" gracePeriod=30
Oct 30 04:13:50.397494 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.397439    4334 generic.go:356] "Generic (PLEG): container finished" podID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerID="33866d725fe7da350960c12cc516e1f8a9470412726525eca9df194ff90fe8d0" exitCode=0
Oct 30 04:13:50.397494 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.397487    4334 generic.go:356] "Generic (PLEG): container finished" podID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerID="94e3232fe7887b1ab1c8cb3d1d509e5e3ad5dde4aa0a4f3c95cd83989b2e97ec" exitCode=0
Oct 30 04:13:50.397494 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.397498    4334 generic.go:356] "Generic (PLEG): container finished" podID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerID="76345f3f31b0d41a7830c03e6e9a3e1d4702009940a4d253a1102e57fd4815d3" exitCode=0
Oct 30 04:13:50.397494 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.397493    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerDied","Data":"33866d725fe7da350960c12cc516e1f8a9470412726525eca9df194ff90fe8d0"}
Oct 30 04:13:50.397843 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.397564    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerDied","Data":"94e3232fe7887b1ab1c8cb3d1d509e5e3ad5dde4aa0a4f3c95cd83989b2e97ec"}
Oct 30 04:13:50.397843 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.397599    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerDied","Data":"76345f3f31b0d41a7830c03e6e9a3e1d4702009940a4d253a1102e57fd4815d3"}
Oct 30 04:13:50.702186 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.702156    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:50.847596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847516    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-kube-api-access-kx8j7\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-kube-api-access-kx8j7\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.847596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847553    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.847596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847584    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.847845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847615    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config-out\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.847845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847723    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.847845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847758    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-thanos-prometheus-http-client-file\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.847845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847789    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.847845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847831    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.848131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847878    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.848131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847923    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.848131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847926    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-1" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:50.848131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.847984    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-secret-prometheus-web-tls-ca\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.848131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.848028    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-tls-assets\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-tls-assets\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.848131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.848067    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config\") pod \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\" (UID: \"79fe0ee4-679d-4a13-82a4-a01a814bc3f6\") "
Oct 30 04:13:50.848473 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.848273    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-2" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:50.848473 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.848380    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.848473 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.848402    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.848935 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.848886    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-0" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:50.850488 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.850464    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config-out" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:13:50.850839 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.850804    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:50.850928 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.850845    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-kube-api-access-kx8j7" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "kube-api-access-kx8j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:50.851124 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.851102    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:50.851213 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.851192    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-secret-prometheus-web-tls-ca" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:50.851782 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.851758    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:50.851888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.851866    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-thanos-prometheus-http-client-file" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:50.852575 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.852556    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-tls-assets" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:50.853161 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.853145    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:13:50.862674 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.862655    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config" pod "79fe0ee4-679d-4a13-82a4-a01a814bc3f6" (UID: "79fe0ee4-679d-4a13-82a4-a01a814bc3f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:50.949319 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949281    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949319 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949320    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx8j7\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-kube-api-access-kx8j7\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949340    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949359    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949405    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949424    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949440    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949472    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949487    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949503    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.949539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.949518    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79fe0ee4-679d-4a13-82a4-a01a814bc3f6-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:50.966177 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:50.966158    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:13:50.980313 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:13:50.980283    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:13:51.050315 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.050293    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\""
Oct 30 04:13:51.405339 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.405317    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:51.405478 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.405316    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"79fe0ee4-679d-4a13-82a4-a01a814bc3f6","Type":"ContainerDied","Data":"5fde79b4fbf6b125ca2f8b1448573bab0edca7ab2b4bc5028fc9870f5bcfd829"}
Oct 30 04:13:51.405531 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.405493    4334 scope.go:122] "RemoveContainer" containerID="33866d725fe7da350960c12cc516e1f8a9470412726525eca9df194ff90fe8d0"
Oct 30 04:13:51.438382 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.438357    4334 scope.go:122] "RemoveContainer" containerID="94e3232fe7887b1ab1c8cb3d1d509e5e3ad5dde4aa0a4f3c95cd83989b2e97ec"
Oct 30 04:13:51.448341 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.448315    4334 scope.go:122] "RemoveContainer" containerID="76345f3f31b0d41a7830c03e6e9a3e1d4702009940a4d253a1102e57fd4815d3"
Oct 30 04:13:51.456169 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.456146    4334 scope.go:122] "RemoveContainer" containerID="228b1199df726ad3dcfe8c087aff71a022d7a721ede33b8cf4b6bde340f00a34"
Oct 30 04:13:51.481620 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.481591    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:13:51.492108 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.492086    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:13:51.539730 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.539706    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:51.654755 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.654721    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-data-science-collector-ta-client-cert\") pod \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") "
Oct 30 04:13:51.654912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.654792    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otc-internal\") pod \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") "
Oct 30 04:13:51.654912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.654812    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-monitoring-tls-certs\") pod \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") "
Oct 30 04:13:51.654912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.654871    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-kube-api-access-kn9x7\" (UniqueName: \"kubernetes.io/projected/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-kube-api-access-kn9x7\") pod \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") "
Oct 30 04:13:51.654912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.654894    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-tls-certs\") pod \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") "
Oct 30 04:13:51.655143 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.654925    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otlp-tls-certs\") pod \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\" (UID: \"0a37aa83-e184-4b1c-bc14-09c9088a4d3e\") "
Oct 30 04:13:51.655276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.655231    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otc-internal" pod "0a37aa83-e184-4b1c-bc14-09c9088a4d3e" (UID: "0a37aa83-e184-4b1c-bc14-09c9088a4d3e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:51.657262 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.657238    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-data-science-collector-ta-client-cert" pod "0a37aa83-e184-4b1c-bc14-09c9088a4d3e" (UID: "0a37aa83-e184-4b1c-bc14-09c9088a4d3e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:51.657375 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.657346    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otlp-tls-certs" pod "0a37aa83-e184-4b1c-bc14-09c9088a4d3e" (UID: "0a37aa83-e184-4b1c-bc14-09c9088a4d3e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:51.657578 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.657552    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-kube-api-access-kn9x7" pod "0a37aa83-e184-4b1c-bc14-09c9088a4d3e" (UID: "0a37aa83-e184-4b1c-bc14-09c9088a4d3e"). InnerVolumeSpecName "kube-api-access-kn9x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:51.657578 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.657563    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-tls-certs" pod "0a37aa83-e184-4b1c-bc14-09c9088a4d3e" (UID: "0a37aa83-e184-4b1c-bc14-09c9088a4d3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:51.657664 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.657571    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-monitoring-tls-certs" pod "0a37aa83-e184-4b1c-bc14-09c9088a4d3e" (UID: "0a37aa83-e184-4b1c-bc14-09c9088a4d3e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:51.755786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.755711    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn9x7\" (UniqueName: \"kubernetes.io/projected/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-kube-api-access-kn9x7\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:51.755786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.755739    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:51.755786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.755754    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:51.755786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.755768    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:51.755786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.755781    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:51.756037 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:51.755793    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a37aa83-e184-4b1c-bc14-09c9088a4d3e-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:52.411490 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.411434    4334 generic.go:356] "Generic (PLEG): container finished" podID="0a37aa83-e184-4b1c-bc14-09c9088a4d3e" containerID="89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7" exitCode=0
Oct 30 04:13:52.412027 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.411495    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.412027 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.411517    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0a37aa83-e184-4b1c-bc14-09c9088a4d3e","Type":"ContainerDied","Data":"89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7"}
Oct 30 04:13:52.412027 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.411569    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0a37aa83-e184-4b1c-bc14-09c9088a4d3e","Type":"ContainerDied","Data":"69aaa20e62442c87b363d4cac7e0936c466146c2ba66e89032e30aed17d06079"}
Oct 30 04:13:52.412027 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.411591    4334 scope.go:122] "RemoveContainer" containerID="89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7"
Oct 30 04:13:52.420671 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.420654    4334 scope.go:122] "RemoveContainer" containerID="89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7"
Oct 30 04:13:52.420921 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:13:52.420902    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7\": container with ID starting with 89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7 not found: ID does not exist" containerID="89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7"
Oct 30 04:13:52.420983 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.420932    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7"} err="failed to get container status \"89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7\": rpc error: code = NotFound desc = could not find container \"89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7\": container with ID starting with 89a426a78f200a0d4d4580fdc6df3650f58fc0047619e86827e901b35f8b6ee7 not found: ID does not exist"
Oct 30 04:13:52.456979 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.456956    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:52.468212 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.468191    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:52.499016 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.498995    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:52.499323 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499312    4334 cpu_manager.go:414] "containerMap: removing container" podUID="0a37aa83-e184-4b1c-bc14-09c9088a4d3e" containerName="otc-container"
Oct 30 04:13:52.499363 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499324    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a37aa83-e184-4b1c-bc14-09c9088a4d3e" containerName="otc-container"
Oct 30 04:13:52.499363 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499339    4334 cpu_manager.go:414] "containerMap: removing container" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="config-reloader"
Oct 30 04:13:52.499363 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499345    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="config-reloader"
Oct 30 04:13:52.499363 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499352    4334 cpu_manager.go:414] "containerMap: removing container" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="init-config-reloader"
Oct 30 04:13:52.499363 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499357    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="init-config-reloader"
Oct 30 04:13:52.499363 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499364    4334 cpu_manager.go:414] "containerMap: removing container" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="prometheus"
Oct 30 04:13:52.499588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499369    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="prometheus"
Oct 30 04:13:52.499588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499377    4334 cpu_manager.go:414] "containerMap: removing container" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="thanos-sidecar"
Oct 30 04:13:52.499588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499382    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="thanos-sidecar"
Oct 30 04:13:52.499588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499437    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a37aa83-e184-4b1c-bc14-09c9088a4d3e" containerName="otc-container"
Oct 30 04:13:52.499588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499457    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="config-reloader"
Oct 30 04:13:52.499588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499464    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="prometheus"
Oct 30 04:13:52.499588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.499470    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" containerName="thanos-sidecar"
Oct 30 04:13:52.503970 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.503945    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.518531 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.518514    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:13:52.518630 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.518568    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:13:52.518716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.518663    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:13:52.519922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.519898    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:52.529002 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.528982    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:13:52.529002 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.528983    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:13:52.529143 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.528989    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-txvtm\""
Oct 30 04:13:52.662961 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.662890    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.662961 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.662923    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.663134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.662961    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfgg\" (UniqueName: \"kubernetes.io/projected/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-kube-api-access-srfgg\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.663134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.663006    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.663134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.663078    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.663134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.663093    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.764311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.764283    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.764489 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.764318    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.764489 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.764349    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfgg\" (UniqueName: \"kubernetes.io/projected/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-kube-api-access-srfgg\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.764489 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.764470    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.764672 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.764528    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.764672 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.764547    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.765235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.765214    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.766858 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.766831    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.766963 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.766946    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.767323 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.767303    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.767362 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.767335    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.783080 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.783057    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfgg\" (UniqueName: \"kubernetes.io/projected/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-kube-api-access-srfgg\") pod \"data-science-collector-collector-1\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:52.816859 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:52.816829    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:53.111475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:53.111324    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:53.115421 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:13:53.115395    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a63d7a_3e33_4a83_98d3_534a3fa8c51e.slice/crio-2518a490f0b2cf5f784ea5ba94be117327cd998811f0a06e73571694e46e3c26 WatchSource:0}: Error finding container 2518a490f0b2cf5f784ea5ba94be117327cd998811f0a06e73571694e46e3c26: Status 404 returned error can't find the container with id 2518a490f0b2cf5f784ea5ba94be117327cd998811f0a06e73571694e46e3c26
Oct 30 04:13:53.117649 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:53.117615    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:13:53.117749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:53.117675    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:13:53.159258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:53.159232    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a37aa83-e184-4b1c-bc14-09c9088a4d3e" path="/var/lib/kubelet/pods/0a37aa83-e184-4b1c-bc14-09c9088a4d3e/volumes"
Oct 30 04:13:53.159686 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:53.159672    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fe0ee4-679d-4a13-82a4-a01a814bc3f6" path="/var/lib/kubelet/pods/79fe0ee4-679d-4a13-82a4-a01a814bc3f6/volumes"
Oct 30 04:13:53.419390 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:53.419351    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"52a63d7a-3e33-4a83-98d3-534a3fa8c51e","Type":"ContainerStarted","Data":"3682d5eddf1fe5c84f93048bca9f269a4b8f7a3e60a878b375cb791256008658"}
Oct 30 04:13:53.419828 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:53.419427    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"52a63d7a-3e33-4a83-98d3-534a3fa8c51e","Type":"ContainerStarted","Data":"2518a490f0b2cf5f784ea5ba94be117327cd998811f0a06e73571694e46e3c26"}
Oct 30 04:13:53.457955 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:53.457902    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.457889426 podStartE2EDuration="1.457889426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:53.455556311 +0000 UTC m=+5512.462146556" watchObservedRunningTime="2026-10-30 04:13:53.457889426 +0000 UTC m=+5512.464479680"
Oct 30 04:13:57.786590 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.786555    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:13:57.790394 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.790374    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.805507 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.805483    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:13:57.805663 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.805500    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:13:57.805794 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.805503    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:13:57.816799 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.816767    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:13:57.816799 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.816775    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:13:57.816799 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.816773    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-fw4nx\""
Oct 30 04:13:57.817060 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.816942    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:13:57.818917 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.818886    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:13:57.882107 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.882074    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:13:57.882107 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.882090    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:13:57.882313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.882301    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:13:57.909194 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909151    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909306 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909204    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909306 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909275    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909306 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909300    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfxx\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-kube-api-access-nlfxx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909345    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909365    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909550    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909586    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909616    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909664    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909806 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909709    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:57.909806 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:57.909752    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010524 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010487    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010696 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010529    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010696 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010575    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010696 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010615    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010696 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010656    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010696 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010678    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010709    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010748    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010784    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.010965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010815    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.011196 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.010965    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.011196 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.011005    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfxx\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-kube-api-access-nlfxx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.011386 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.011356    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.012136 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.012109    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.012337 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.012189    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.013796 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.013747    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.013984 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.013958    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.014189 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.014156    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.014305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.014192    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.014305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.014284    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.014410 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.014321    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.014769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.014748    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.015221 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.015206    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.028439 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.028414    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfxx\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-kube-api-access-nlfxx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:13:58.246922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.246876    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:58.247136 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.247110    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="52a63d7a-3e33-4a83-98d3-534a3fa8c51e" containerName="otc-container" containerID="cri-o://3682d5eddf1fe5c84f93048bca9f269a4b8f7a3e60a878b375cb791256008658" gracePeriod=30
Oct 30 04:13:58.446707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.446675    4334 generic.go:356] "Generic (PLEG): container finished" podID="52a63d7a-3e33-4a83-98d3-534a3fa8c51e" containerID="3682d5eddf1fe5c84f93048bca9f269a4b8f7a3e60a878b375cb791256008658" exitCode=0
Oct 30 04:13:58.446855 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.446730    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"52a63d7a-3e33-4a83-98d3-534a3fa8c51e","Type":"ContainerDied","Data":"3682d5eddf1fe5c84f93048bca9f269a4b8f7a3e60a878b375cb791256008658"}
Oct 30 04:13:58.493971 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.493945    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:58.615867 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.615789    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otc-internal\") pod \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") "
Oct 30 04:13:58.615867 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.615850    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-monitoring-tls-certs\") pod \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") "
Oct 30 04:13:58.616102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.615871    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otlp-tls-certs\") pod \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") "
Oct 30 04:13:58.616102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.615904    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-kube-api-access-srfgg\" (UniqueName: \"kubernetes.io/projected/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-kube-api-access-srfgg\") pod \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") "
Oct 30 04:13:58.616102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.615934    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-tls-certs\") pod \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") "
Oct 30 04:13:58.616102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.615974    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-data-science-collector-ta-client-cert\") pod \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\" (UID: \"52a63d7a-3e33-4a83-98d3-534a3fa8c51e\") "
Oct 30 04:13:58.616303 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.616254    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otc-internal" pod "52a63d7a-3e33-4a83-98d3-534a3fa8c51e" (UID: "52a63d7a-3e33-4a83-98d3-534a3fa8c51e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:58.618054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.618023    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-monitoring-tls-certs" pod "52a63d7a-3e33-4a83-98d3-534a3fa8c51e" (UID: "52a63d7a-3e33-4a83-98d3-534a3fa8c51e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:58.618187 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.618152    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-kube-api-access-srfgg" pod "52a63d7a-3e33-4a83-98d3-534a3fa8c51e" (UID: "52a63d7a-3e33-4a83-98d3-534a3fa8c51e"). InnerVolumeSpecName "kube-api-access-srfgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:58.618187 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.618164    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-tls-certs" pod "52a63d7a-3e33-4a83-98d3-534a3fa8c51e" (UID: "52a63d7a-3e33-4a83-98d3-534a3fa8c51e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:58.618553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.618534    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-data-science-collector-ta-client-cert" pod "52a63d7a-3e33-4a83-98d3-534a3fa8c51e" (UID: "52a63d7a-3e33-4a83-98d3-534a3fa8c51e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:58.618616 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.618549    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otlp-tls-certs" pod "52a63d7a-3e33-4a83-98d3-534a3fa8c51e" (UID: "52a63d7a-3e33-4a83-98d3-534a3fa8c51e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:58.717249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.717217    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:58.717249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.717244    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:58.717420 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.717257    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:58.717420 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.717271    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:58.717420 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.717283    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srfgg\" (UniqueName: \"kubernetes.io/projected/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-kube-api-access-srfgg\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:58.717420 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:58.717293    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a63d7a-3e33-4a83-98d3-534a3fa8c51e-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:13:59.452995 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.452969    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.453411 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.452971    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"52a63d7a-3e33-4a83-98d3-534a3fa8c51e","Type":"ContainerDied","Data":"2518a490f0b2cf5f784ea5ba94be117327cd998811f0a06e73571694e46e3c26"}
Oct 30 04:13:59.453411 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.453086    4334 scope.go:122] "RemoveContainer" containerID="3682d5eddf1fe5c84f93048bca9f269a4b8f7a3e60a878b375cb791256008658"
Oct 30 04:13:59.500076 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.500050    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:59.511126 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.511105    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:59.538500 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.538473    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:59.538811 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.538797    4334 cpu_manager.go:414] "containerMap: removing container" podUID="52a63d7a-3e33-4a83-98d3-534a3fa8c51e" containerName="otc-container"
Oct 30 04:13:59.538811 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.538810    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a63d7a-3e33-4a83-98d3-534a3fa8c51e" containerName="otc-container"
Oct 30 04:13:59.538917 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.538877    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a63d7a-3e33-4a83-98d3-534a3fa8c51e" containerName="otc-container"
Oct 30 04:13:59.543541 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.543517    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.559200 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.559175    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:13:59.559978 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.559922    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-txvtm\""
Oct 30 04:13:59.560084 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.560022    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:13:59.561417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.561390    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:13:59.570026 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.570000    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:13:59.570176 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.570028    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:13:59.570176 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.570034    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:13:59.625086 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.625054    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.625086 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.625087    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.625330 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.625107    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.625330 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.625223    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.625330 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.625246    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phv72\" (UniqueName: \"kubernetes.io/projected/37281846-54ba-413b-8b7c-ac7cbc1af6b6-kube-api-access-phv72\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.625330 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.625288    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.726043 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.725972    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phv72\" (UniqueName: \"kubernetes.io/projected/37281846-54ba-413b-8b7c-ac7cbc1af6b6-kube-api-access-phv72\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.726179 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.726048    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.726179 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.726087    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.726179 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.726114    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.726179 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.726143    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.726373 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.726207    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.726885 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.726857    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.728782 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.728755    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.728906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.728886    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.728974 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.728924    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.728974 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.728939    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.744422 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.744397    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phv72\" (UniqueName: \"kubernetes.io/projected/37281846-54ba-413b-8b7c-ac7cbc1af6b6-kube-api-access-phv72\") pod \"data-science-collector-collector-1\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:13:59.855570 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:13:59.855544    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:00.150939 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:00.150869    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:00.154511 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:14:00.154478    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37281846_54ba_413b_8b7c_ac7cbc1af6b6.slice/crio-809986a195cffc3e2d22c706817462166ca1cf43caf7361442ef7b8fcac16cf0 WatchSource:0}: Error finding container 809986a195cffc3e2d22c706817462166ca1cf43caf7361442ef7b8fcac16cf0: Status 404 returned error can't find the container with id 809986a195cffc3e2d22c706817462166ca1cf43caf7361442ef7b8fcac16cf0
Oct 30 04:14:00.157074 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:00.157034    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:14:00.157166 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:00.157105    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:14:00.460061 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:00.460029    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"37281846-54ba-413b-8b7c-ac7cbc1af6b6","Type":"ContainerStarted","Data":"82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5"}
Oct 30 04:14:00.460571 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:00.460077    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"37281846-54ba-413b-8b7c-ac7cbc1af6b6","Type":"ContainerStarted","Data":"809986a195cffc3e2d22c706817462166ca1cf43caf7361442ef7b8fcac16cf0"}
Oct 30 04:14:00.497670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:00.497626    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.497613703 podStartE2EDuration="1.497613703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:00.49667977 +0000 UTC m=+5519.503270024" watchObservedRunningTime="2026-10-30 04:14:00.497613703 +0000 UTC m=+5519.504203957"
Oct 30 04:14:01.158869 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:01.158840    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a63d7a-3e33-4a83-98d3-534a3fa8c51e" path="/var/lib/kubelet/pods/52a63d7a-3e33-4a83-98d3-534a3fa8c51e/volumes"
Oct 30 04:14:06.278588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.278554    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:06.304680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.304652    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:06.379162 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.379133    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:06.379307 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.379260    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:06.392289 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.392261    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:06.492196 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.492168    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:06.519112 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.519090    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:06.824276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.824248    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-fw4nx\""
Oct 30 04:14:06.832337 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:06.832311    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:07.158827 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:07.158795    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:14:07.162419 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:14:07.162395    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2909166b_a7a1_4528_b8c9_a83ba8d7ad67.slice/crio-44787b4cfba24484e02b9c5b28656b74c4744da283c6d99a0ab535e7939f8e3f WatchSource:0}: Error finding container 44787b4cfba24484e02b9c5b28656b74c4744da283c6d99a0ab535e7939f8e3f: Status 404 returned error can't find the container with id 44787b4cfba24484e02b9c5b28656b74c4744da283c6d99a0ab535e7939f8e3f
Oct 30 04:14:07.492800 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:07.492720    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerStarted","Data":"44787b4cfba24484e02b9c5b28656b74c4744da283c6d99a0ab535e7939f8e3f"}
Oct 30 04:14:08.499749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:08.499714    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerStarted","Data":"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"}
Oct 30 04:14:11.515307 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:11.515231    4334 generic.go:356] "Generic (PLEG): container finished" podID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerID="4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38" exitCode=0
Oct 30 04:14:11.515307 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:11.515291    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerDied","Data":"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"}
Oct 30 04:14:12.522912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:12.522878    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerStarted","Data":"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"}
Oct 30 04:14:12.523307 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:12.522967    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerStarted","Data":"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"}
Oct 30 04:14:12.523307 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:12.522995    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerStarted","Data":"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"}
Oct 30 04:14:12.591055 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:12.591002    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=12.005223645 podStartE2EDuration="15.590988994s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.585765349s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:12.589509981 +0000 UTC m=+5531.596100235" watchObservedRunningTime="2026-10-30 04:14:12.590988994 +0000 UTC m=+5531.597579267"
Oct 30 04:14:16.832945 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:16.832914    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:20.978501 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:20.978386    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:14:21.833169 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:21.833136    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:21.838628 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:21.838604    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:22.577566 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:22.577540    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:29.947128 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:29.947092    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:29.947647 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:29.947342    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="37281846-54ba-413b-8b7c-ac7cbc1af6b6" containerName="otc-container" containerID="cri-o://82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5" gracePeriod=30
Oct 30 04:14:32.289364 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.289339    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.386438 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.386363    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-data-science-collector-ta-client-cert\") pod \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") "
Oct 30 04:14:32.386438 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.386436    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-monitoring-tls-certs\") pod \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") "
Oct 30 04:14:32.386687 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.386482    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-tls-certs\") pod \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") "
Oct 30 04:14:32.386687 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.386507    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otc-internal\") pod \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") "
Oct 30 04:14:32.386687 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.386530    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37281846-54ba-413b-8b7c-ac7cbc1af6b6-kube-api-access-phv72\" (UniqueName: \"kubernetes.io/projected/37281846-54ba-413b-8b7c-ac7cbc1af6b6-kube-api-access-phv72\") pod \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") "
Oct 30 04:14:32.386687 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.386561    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otlp-tls-certs\") pod \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\" (UID: \"37281846-54ba-413b-8b7c-ac7cbc1af6b6\") "
Oct 30 04:14:32.386968 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.386908    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otc-internal" pod "37281846-54ba-413b-8b7c-ac7cbc1af6b6" (UID: "37281846-54ba-413b-8b7c-ac7cbc1af6b6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:32.388790 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.388747    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-data-science-collector-ta-client-cert" pod "37281846-54ba-413b-8b7c-ac7cbc1af6b6" (UID: "37281846-54ba-413b-8b7c-ac7cbc1af6b6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:32.388924 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.388835    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-tls-certs" pod "37281846-54ba-413b-8b7c-ac7cbc1af6b6" (UID: "37281846-54ba-413b-8b7c-ac7cbc1af6b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:32.388924 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.388888    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otlp-tls-certs" pod "37281846-54ba-413b-8b7c-ac7cbc1af6b6" (UID: "37281846-54ba-413b-8b7c-ac7cbc1af6b6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:32.389162 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.389138    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-monitoring-tls-certs" pod "37281846-54ba-413b-8b7c-ac7cbc1af6b6" (UID: "37281846-54ba-413b-8b7c-ac7cbc1af6b6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:32.389265 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.389191    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37281846-54ba-413b-8b7c-ac7cbc1af6b6-kube-api-access-phv72" pod "37281846-54ba-413b-8b7c-ac7cbc1af6b6" (UID: "37281846-54ba-413b-8b7c-ac7cbc1af6b6"). InnerVolumeSpecName "kube-api-access-phv72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:32.487780 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.487749    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:32.487780 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.487775    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:32.487780 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.487786    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:32.488000 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.487795    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phv72\" (UniqueName: \"kubernetes.io/projected/37281846-54ba-413b-8b7c-ac7cbc1af6b6-kube-api-access-phv72\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:32.488000 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.487803    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:32.488000 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.487814    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/37281846-54ba-413b-8b7c-ac7cbc1af6b6-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:32.620529 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.620497    4334 generic.go:356] "Generic (PLEG): container finished" podID="37281846-54ba-413b-8b7c-ac7cbc1af6b6" containerID="82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5" exitCode=0
Oct 30 04:14:32.620691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.620553    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.620691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.620584    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"37281846-54ba-413b-8b7c-ac7cbc1af6b6","Type":"ContainerDied","Data":"82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5"}
Oct 30 04:14:32.620691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.620664    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"37281846-54ba-413b-8b7c-ac7cbc1af6b6","Type":"ContainerDied","Data":"809986a195cffc3e2d22c706817462166ca1cf43caf7361442ef7b8fcac16cf0"}
Oct 30 04:14:32.620691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.620687    4334 scope.go:122] "RemoveContainer" containerID="82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5"
Oct 30 04:14:32.629698 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.629682    4334 scope.go:122] "RemoveContainer" containerID="82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5"
Oct 30 04:14:32.629958 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:32.629936    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5\": container with ID starting with 82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5 not found: ID does not exist" containerID="82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5"
Oct 30 04:14:32.630059 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.629964    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5"} err="failed to get container status \"82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5\": rpc error: code = NotFound desc = could not find container \"82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5\": container with ID starting with 82e940bed260f889c3975375dceab887cadd8bf2cc0416697830a12a76ae59e5 not found: ID does not exist"
Oct 30 04:14:32.667012 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.666991    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:32.672277 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.672259    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:32.707488 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.707468    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:32.707825 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.707812    4334 cpu_manager.go:414] "containerMap: removing container" podUID="37281846-54ba-413b-8b7c-ac7cbc1af6b6" containerName="otc-container"
Oct 30 04:14:32.707878 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.707826    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37281846-54ba-413b-8b7c-ac7cbc1af6b6" containerName="otc-container"
Oct 30 04:14:32.707915 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.707910    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="37281846-54ba-413b-8b7c-ac7cbc1af6b6" containerName="otc-container"
Oct 30 04:14:32.712329 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.712315    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.728676 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.728655    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:14:32.728923 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.728894    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:14:32.728923 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.728894    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:14:32.729092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.728972    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-txvtm\""
Oct 30 04:14:32.729092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.729070    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:14:32.729516 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.729497    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:14:32.730938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.730919    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:32.790643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.790607    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.790643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.790643    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9t6w\" (UniqueName: \"kubernetes.io/projected/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-kube-api-access-d9t6w\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.790947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.790792    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.790947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.790823    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.790947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.790852    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.791060 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.790949    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.892234 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.892206    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.892234 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.892237    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9t6w\" (UniqueName: \"kubernetes.io/projected/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-kube-api-access-d9t6w\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.892426 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.892273    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.892426 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.892291    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.892426 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.892311    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.892426 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.892337    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.893224 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.893195    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.894882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.894856    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.895006 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.894914    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.895006 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.894967    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.895093 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.895014    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:32.909489 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:32.909470    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9t6w\" (UniqueName: \"kubernetes.io/projected/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-kube-api-access-d9t6w\") pod \"data-science-collector-collector-1\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:33.024193 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:33.024134    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:33.158962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:33.158933    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37281846-54ba-413b-8b7c-ac7cbc1af6b6" path="/var/lib/kubelet/pods/37281846-54ba-413b-8b7c-ac7cbc1af6b6/volumes"
Oct 30 04:14:33.319349 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:33.319253    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:33.325343 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:14:33.325316    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcc8e6d_421d_47de_8659_87fbf1dcbb28.slice/crio-510d13454af3c61df513a6d7c1cd602b84f1121e6a4c4b7dc97989d8fe2acc48 WatchSource:0}: Error finding container 510d13454af3c61df513a6d7c1cd602b84f1121e6a4c4b7dc97989d8fe2acc48: Status 404 returned error can't find the container with id 510d13454af3c61df513a6d7c1cd602b84f1121e6a4c4b7dc97989d8fe2acc48
Oct 30 04:14:33.327499 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:33.327464    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:14:33.327598 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:33.327524    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:14:33.629145 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:33.629070    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6fcc8e6d-421d-47de-8659-87fbf1dcbb28","Type":"ContainerStarted","Data":"98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552"}
Oct 30 04:14:33.629145 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:33.629124    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6fcc8e6d-421d-47de-8659-87fbf1dcbb28","Type":"ContainerStarted","Data":"510d13454af3c61df513a6d7c1cd602b84f1121e6a4c4b7dc97989d8fe2acc48"}
Oct 30 04:14:33.668206 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:33.668143    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6681247479999999 podStartE2EDuration="1.668124748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:33.665171048 +0000 UTC m=+5552.671761302" watchObservedRunningTime="2026-10-30 04:14:33.668124748 +0000 UTC m=+5552.674715002"
Oct 30 04:14:34.290422 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.290391    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"]
Oct 30 04:14:34.290607 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.290592    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerName="kube-rbac-proxy" containerID="cri-o://695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0" gracePeriod=30
Oct 30 04:14:34.290710 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.290679    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerName="prom-label-proxy" containerID="cri-o://647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7" gracePeriod=30
Oct 30 04:14:34.309712 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.309685    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"]
Oct 30 04:14:34.309926 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.309907    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerName="kube-rbac-proxy" containerID="cri-o://5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211" gracePeriod=30
Oct 30 04:14:34.309972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.309934    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerName="prom-label-proxy" containerID="cri-o://c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f" gracePeriod=30
Oct 30 04:14:34.550757 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.550705    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:14:34.572326 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.572307    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:14:34.607852 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.607823    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-kube-api-access-2f267\" (UniqueName: \"kubernetes.io/projected/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-kube-api-access-2f267\") pod \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") "
Oct 30 04:14:34.608308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.607882    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-client-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-client-ca\") pod \"681c1cde-4758-4c4b-8d76-3105831c2f21\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") "
Oct 30 04:14:34.608308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.607927    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681c1cde-4758-4c4b-8d76-3105831c2f21-kube-api-access-4w5z6\" (UniqueName: \"kubernetes.io/projected/681c1cde-4758-4c4b-8d76-3105831c2f21-kube-api-access-4w5z6\") pod \"681c1cde-4758-4c4b-8d76-3105831c2f21\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") "
Oct 30 04:14:34.608308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.607964    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-config\" (UniqueName: \"kubernetes.io/configmap/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-config\") pod \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") "
Oct 30 04:14:34.608308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.608011    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-prometheus-tls-ca\") pod \"681c1cde-4758-4c4b-8d76-3105831c2f21\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") "
Oct 30 04:14:34.608308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.608039    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/681c1cde-4758-4c4b-8d76-3105831c2f21-tls\" (UniqueName: \"kubernetes.io/secret/681c1cde-4758-4c4b-8d76-3105831c2f21-tls\") pod \"681c1cde-4758-4c4b-8d76-3105831c2f21\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") "
Oct 30 04:14:34.608308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.608087    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-config\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-config\") pod \"681c1cde-4758-4c4b-8d76-3105831c2f21\" (UID: \"681c1cde-4758-4c4b-8d76-3105831c2f21\") "
Oct 30 04:14:34.608308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.608122    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-tls\" (UniqueName: \"kubernetes.io/secret/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-tls\") pod \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\" (UID: \"4d3b9a83-b96c-439c-9aaa-34895adb3c7a\") "
Oct 30 04:14:34.608970 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.608939    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-prometheus-tls-ca" pod "681c1cde-4758-4c4b-8d76-3105831c2f21" (UID: "681c1cde-4758-4c4b-8d76-3105831c2f21"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:34.609951 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.609856    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-client-ca" pod "681c1cde-4758-4c4b-8d76-3105831c2f21" (UID: "681c1cde-4758-4c4b-8d76-3105831c2f21"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:34.610269 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.610244    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-config" pod "681c1cde-4758-4c4b-8d76-3105831c2f21" (UID: "681c1cde-4758-4c4b-8d76-3105831c2f21"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:34.610805 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.610775    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-config" pod "4d3b9a83-b96c-439c-9aaa-34895adb3c7a" (UID: "4d3b9a83-b96c-439c-9aaa-34895adb3c7a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:34.611780 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.611755    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-kube-api-access-2f267" pod "4d3b9a83-b96c-439c-9aaa-34895adb3c7a" (UID: "4d3b9a83-b96c-439c-9aaa-34895adb3c7a"). InnerVolumeSpecName "kube-api-access-2f267". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:34.613471 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.613306    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-tls" pod "4d3b9a83-b96c-439c-9aaa-34895adb3c7a" (UID: "4d3b9a83-b96c-439c-9aaa-34895adb3c7a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:34.615069 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.615048    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681c1cde-4758-4c4b-8d76-3105831c2f21-kube-api-access-4w5z6" pod "681c1cde-4758-4c4b-8d76-3105831c2f21" (UID: "681c1cde-4758-4c4b-8d76-3105831c2f21"). InnerVolumeSpecName "kube-api-access-4w5z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:34.615149 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.615075    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681c1cde-4758-4c4b-8d76-3105831c2f21-tls" pod "681c1cde-4758-4c4b-8d76-3105831c2f21" (UID: "681c1cde-4758-4c4b-8d76-3105831c2f21"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:34.642536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.642506    4334 generic.go:356] "Generic (PLEG): container finished" podID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerID="647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7" exitCode=0
Oct 30 04:14:34.642536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.642530    4334 generic.go:356] "Generic (PLEG): container finished" podID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerID="695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0" exitCode=0
Oct 30 04:14:34.642762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.642562    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"
Oct 30 04:14:34.642762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.642594    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" event={"ID":"681c1cde-4758-4c4b-8d76-3105831c2f21","Type":"ContainerDied","Data":"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7"}
Oct 30 04:14:34.642762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.642667    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" event={"ID":"681c1cde-4758-4c4b-8d76-3105831c2f21","Type":"ContainerDied","Data":"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0"}
Oct 30 04:14:34.642762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.642706    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r" event={"ID":"681c1cde-4758-4c4b-8d76-3105831c2f21","Type":"ContainerDied","Data":"df7d610e9ecf8568ca7cbfaf2903b2706cb0c865b945d58c9a56426115196565"}
Oct 30 04:14:34.642762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.642746    4334 scope.go:122] "RemoveContainer" containerID="647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7"
Oct 30 04:14:34.644433 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.644406    4334 generic.go:356] "Generic (PLEG): container finished" podID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerID="c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f" exitCode=0
Oct 30 04:14:34.644433 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.644434    4334 generic.go:356] "Generic (PLEG): container finished" podID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerID="5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211" exitCode=0
Oct 30 04:14:34.644648 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.644485    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"
Oct 30 04:14:34.644648 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.644485    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" event={"ID":"4d3b9a83-b96c-439c-9aaa-34895adb3c7a","Type":"ContainerDied","Data":"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f"}
Oct 30 04:14:34.644772 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.644645    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" event={"ID":"4d3b9a83-b96c-439c-9aaa-34895adb3c7a","Type":"ContainerDied","Data":"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211"}
Oct 30 04:14:34.644772 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.644680    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk" event={"ID":"4d3b9a83-b96c-439c-9aaa-34895adb3c7a","Type":"ContainerDied","Data":"f4e64c9d908b26e87c9736c4cd5871bea72a94e8fcf2280e4c4b678ea6b06f5e"}
Oct 30 04:14:34.654829 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.654808    4334 scope.go:122] "RemoveContainer" containerID="695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0"
Oct 30 04:14:34.669233 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.669206    4334 scope.go:122] "RemoveContainer" containerID="647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7"
Oct 30 04:14:34.669701 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:34.669681    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7\": container with ID starting with 647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7 not found: ID does not exist" containerID="647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7"
Oct 30 04:14:34.669777 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.669712    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7"} err="failed to get container status \"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7\": rpc error: code = NotFound desc = could not find container \"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7\": container with ID starting with 647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7 not found: ID does not exist"
Oct 30 04:14:34.669777 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.669741    4334 scope.go:122] "RemoveContainer" containerID="695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0"
Oct 30 04:14:34.670051 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:34.670024    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0\": container with ID starting with 695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0 not found: ID does not exist" containerID="695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0"
Oct 30 04:14:34.670377 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.670061    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0"} err="failed to get container status \"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0\": rpc error: code = NotFound desc = could not find container \"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0\": container with ID starting with 695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0 not found: ID does not exist"
Oct 30 04:14:34.670377 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.670083    4334 scope.go:122] "RemoveContainer" containerID="647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7"
Oct 30 04:14:34.670377 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.670361    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7"} err="failed to get container status \"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7\": rpc error: code = NotFound desc = could not find container \"647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7\": container with ID starting with 647a4c4f4cfb5cc5a7e3be4aca0b35d45996ec55ec92201107da46c2c34c0cc7 not found: ID does not exist"
Oct 30 04:14:34.670939 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.670385    4334 scope.go:122] "RemoveContainer" containerID="695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0"
Oct 30 04:14:34.670939 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.670694    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0"} err="failed to get container status \"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0\": rpc error: code = NotFound desc = could not find container \"695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0\": container with ID starting with 695c56cf393e866ea532e99be219aee638106796178071f4394500ebf2da74a0 not found: ID does not exist"
Oct 30 04:14:34.670939 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.670716    4334 scope.go:122] "RemoveContainer" containerID="c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f"
Oct 30 04:14:34.680722 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.680694    4334 scope.go:122] "RemoveContainer" containerID="5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211"
Oct 30 04:14:34.689052 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.689033    4334 scope.go:122] "RemoveContainer" containerID="c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f"
Oct 30 04:14:34.689339 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:34.689322    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f\": container with ID starting with c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f not found: ID does not exist" containerID="c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f"
Oct 30 04:14:34.689386 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.689344    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f"} err="failed to get container status \"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f\": rpc error: code = NotFound desc = could not find container \"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f\": container with ID starting with c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f not found: ID does not exist"
Oct 30 04:14:34.689386 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.689358    4334 scope.go:122] "RemoveContainer" containerID="5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211"
Oct 30 04:14:34.689658 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:34.689640    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211\": container with ID starting with 5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211 not found: ID does not exist" containerID="5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211"
Oct 30 04:14:34.689707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.689664    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211"} err="failed to get container status \"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211\": rpc error: code = NotFound desc = could not find container \"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211\": container with ID starting with 5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211 not found: ID does not exist"
Oct 30 04:14:34.689707 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.689679    4334 scope.go:122] "RemoveContainer" containerID="c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f"
Oct 30 04:14:34.689960 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.689930    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f"} err="failed to get container status \"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f\": rpc error: code = NotFound desc = could not find container \"c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f\": container with ID starting with c6632cf86d7695b64182a4cb833f6bda495cbedc036add06226cb0796bfaff1f not found: ID does not exist"
Oct 30 04:14:34.690045 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.689960    4334 scope.go:122] "RemoveContainer" containerID="5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211"
Oct 30 04:14:34.690219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.690199    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211"} err="failed to get container status \"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211\": rpc error: code = NotFound desc = could not find container \"5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211\": container with ID starting with 5359725accdfa7018fd08c15ac65e51f8b2ff275fff0ac739f95740d39ae2211 not found: ID does not exist"
Oct 30 04:14:34.708376 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.708343    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"]
Oct 30 04:14:34.708989 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.708967    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.709162 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.709146    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f267\" (UniqueName: \"kubernetes.io/projected/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-kube-api-access-2f267\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.709275 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.709260    4334 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-client-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.709394 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.709381    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w5z6\" (UniqueName: \"kubernetes.io/projected/681c1cde-4758-4c4b-8d76-3105831c2f21-kube-api-access-4w5z6\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.709545 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.709530    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d3b9a83-b96c-439c-9aaa-34895adb3c7a-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.709683 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.709656    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-prometheus-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.709683 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.709678    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/681c1cde-4758-4c4b-8d76-3105831c2f21-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.709866 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.709692    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/681c1cde-4758-4c4b-8d76-3105831c2f21-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:34.712731 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.712696    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-vp28r"]
Oct 30 04:14:34.745057 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.744695    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"]
Oct 30 04:14:34.749072 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.749043    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hvspk"]
Oct 30 04:14:34.754295 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.754271    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:14:34.754550 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.754519    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="prometheus" containerID="cri-o://1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d" gracePeriod=600
Oct 30 04:14:34.754550 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.754534    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="thanos-sidecar" containerID="cri-o://a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5" gracePeriod=600
Oct 30 04:14:34.754768 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.754572    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="config-reloader" containerID="cri-o://0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48" gracePeriod=600
Oct 30 04:14:34.771061 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.771031    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:14:34.771273 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.771253    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="alertmanager" containerID="cri-o://7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5" gracePeriod=120
Oct 30 04:14:34.771344 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:34.771324    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="config-reloader" containerID="cri-o://174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086" gracePeriod=120
Oct 30 04:14:35.006859 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.006833    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:35.113730 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.113700    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config-out\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.113947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.113755    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.113947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.113805    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.113947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.113828    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-kube-api-access-nlfxx\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-kube-api-access-nlfxx\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.113947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.113859    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.113947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.113891    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.114236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114004    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.114236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114052    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-secret-prometheus-web-tls-ca\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.114236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114105    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.114236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114153    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.114236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114181    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-tls-assets\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-tls-assets\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.114236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114212    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.114569 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114252    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-thanos-prometheus-http-client-file\") pod \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\" (UID: \"2909166b-a7a1-4528-b8c9-a83ba8d7ad67\") "
Oct 30 04:14:35.114569 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114460    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-0" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:35.114677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114574    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.114677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.114436    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-2" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:35.115434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.115325    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-1" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:35.116871 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.116839    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.117116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.117087    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.117310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.117269    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config-out" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:14:35.117645 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.117559    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-thanos-prometheus-http-client-file" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.117645 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.117564    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-kube-api-access-nlfxx" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "kube-api-access-nlfxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:35.117849 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.117827    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.118286 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.118265    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-tls-assets" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:35.118398 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.118367    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-secret-prometheus-web-tls-ca" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.119627 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.119608    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:14:35.128870 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.128845    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config" pod "2909166b-a7a1-4528-b8c9-a83ba8d7ad67" (UID: "2909166b-a7a1-4528-b8c9-a83ba8d7ad67"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:35.159019 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.158992    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" path="/var/lib/kubelet/pods/4d3b9a83-b96c-439c-9aaa-34895adb3c7a/volumes"
Oct 30 04:14:35.159406 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.159393    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" path="/var/lib/kubelet/pods/681c1cde-4758-4c4b-8d76-3105831c2f21/volumes"
Oct 30 04:14:35.215328 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215303    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlfxx\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-kube-api-access-nlfxx\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215328 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215325    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215336    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215362    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215374    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215388    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215401    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215410    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215418    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215427    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215435    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.215479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.215442    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2909166b-a7a1-4528-b8c9-a83ba8d7ad67-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:35.225965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.225947    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:14:35.316479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.316443    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\""
Oct 30 04:14:35.652415 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.652380    4334 generic.go:356] "Generic (PLEG): container finished" podID="23efc980-dc09-4412-b7cc-439358662567" containerID="174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086" exitCode=0
Oct 30 04:14:35.652829 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.652463    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"23efc980-dc09-4412-b7cc-439358662567","Type":"ContainerDied","Data":"174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086"}
Oct 30 04:14:35.654502 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654474    4334 generic.go:356] "Generic (PLEG): container finished" podID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerID="a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5" exitCode=0
Oct 30 04:14:35.654640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654500    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerDied","Data":"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"}
Oct 30 04:14:35.654640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654497    4334 generic.go:356] "Generic (PLEG): container finished" podID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerID="0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48" exitCode=0
Oct 30 04:14:35.654640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654525    4334 generic.go:356] "Generic (PLEG): container finished" podID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerID="1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d" exitCode=0
Oct 30 04:14:35.654640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654555    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerDied","Data":"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"}
Oct 30 04:14:35.654640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654574    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:14:35.654640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654583    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerDied","Data":"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"}
Oct 30 04:14:35.654640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654607    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2909166b-a7a1-4528-b8c9-a83ba8d7ad67","Type":"ContainerDied","Data":"44787b4cfba24484e02b9c5b28656b74c4744da283c6d99a0ab535e7939f8e3f"}
Oct 30 04:14:35.654640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.654636    4334 scope.go:122] "RemoveContainer" containerID="a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"
Oct 30 04:14:35.663177 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.663159    4334 scope.go:122] "RemoveContainer" containerID="0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"
Oct 30 04:14:35.671464 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.671431    4334 scope.go:122] "RemoveContainer" containerID="1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"
Oct 30 04:14:35.679173 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.679152    4334 scope.go:122] "RemoveContainer" containerID="4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"
Oct 30 04:14:35.687256 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.687238    4334 scope.go:122] "RemoveContainer" containerID="a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"
Oct 30 04:14:35.687577 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:35.687553    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5\": container with ID starting with a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5 not found: ID does not exist" containerID="a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"
Oct 30 04:14:35.687653 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.687583    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"} err="failed to get container status \"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5\": rpc error: code = NotFound desc = could not find container \"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5\": container with ID starting with a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5 not found: ID does not exist"
Oct 30 04:14:35.687653 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.687600    4334 scope.go:122] "RemoveContainer" containerID="0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"
Oct 30 04:14:35.687858 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:35.687841    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48\": container with ID starting with 0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48 not found: ID does not exist" containerID="0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"
Oct 30 04:14:35.687925 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.687868    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"} err="failed to get container status \"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48\": rpc error: code = NotFound desc = could not find container \"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48\": container with ID starting with 0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48 not found: ID does not exist"
Oct 30 04:14:35.687925 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.687890    4334 scope.go:122] "RemoveContainer" containerID="1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"
Oct 30 04:14:35.688148 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:35.688133    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d\": container with ID starting with 1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d not found: ID does not exist" containerID="1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"
Oct 30 04:14:35.688213 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.688154    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"} err="failed to get container status \"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d\": rpc error: code = NotFound desc = could not find container \"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d\": container with ID starting with 1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d not found: ID does not exist"
Oct 30 04:14:35.688213 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.688172    4334 scope.go:122] "RemoveContainer" containerID="4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"
Oct 30 04:14:35.688399 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:35.688383    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38\": container with ID starting with 4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38 not found: ID does not exist" containerID="4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"
Oct 30 04:14:35.688479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.688407    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"} err="failed to get container status \"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38\": rpc error: code = NotFound desc = could not find container \"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38\": container with ID starting with 4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38 not found: ID does not exist"
Oct 30 04:14:35.688479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.688429    4334 scope.go:122] "RemoveContainer" containerID="a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"
Oct 30 04:14:35.688696 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.688669    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"} err="failed to get container status \"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5\": rpc error: code = NotFound desc = could not find container \"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5\": container with ID starting with a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5 not found: ID does not exist"
Oct 30 04:14:35.688696 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.688690    4334 scope.go:122] "RemoveContainer" containerID="0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"
Oct 30 04:14:35.688942 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.688923    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"} err="failed to get container status \"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48\": rpc error: code = NotFound desc = could not find container \"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48\": container with ID starting with 0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48 not found: ID does not exist"
Oct 30 04:14:35.688942 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.688941    4334 scope.go:122] "RemoveContainer" containerID="1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"
Oct 30 04:14:35.689158 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.689141    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"} err="failed to get container status \"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d\": rpc error: code = NotFound desc = could not find container \"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d\": container with ID starting with 1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d not found: ID does not exist"
Oct 30 04:14:35.689158 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.689159    4334 scope.go:122] "RemoveContainer" containerID="4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"
Oct 30 04:14:35.689394 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.689379    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"} err="failed to get container status \"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38\": rpc error: code = NotFound desc = could not find container \"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38\": container with ID starting with 4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38 not found: ID does not exist"
Oct 30 04:14:35.689394 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.689394    4334 scope.go:122] "RemoveContainer" containerID="a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"
Oct 30 04:14:35.689660 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.689633    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5"} err="failed to get container status \"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5\": rpc error: code = NotFound desc = could not find container \"a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5\": container with ID starting with a6feaba898460d0dc7d859953b8434fc0ecaf3ce7252edc123187209d9c5e9e5 not found: ID does not exist"
Oct 30 04:14:35.689713 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.689663    4334 scope.go:122] "RemoveContainer" containerID="0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"
Oct 30 04:14:35.689944 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.689922    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48"} err="failed to get container status \"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48\": rpc error: code = NotFound desc = could not find container \"0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48\": container with ID starting with 0227368d08e8c81ebc463201d7f7c3301b06d0d22ccd00b4ad0cf5fc0dfdec48 not found: ID does not exist"
Oct 30 04:14:35.689991 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.689949    4334 scope.go:122] "RemoveContainer" containerID="1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"
Oct 30 04:14:35.690181 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.690162    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d"} err="failed to get container status \"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d\": rpc error: code = NotFound desc = could not find container \"1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d\": container with ID starting with 1faa1363e318c417a7d38a41a357d1f48beeb360804d96485cd2817e21f1b13d not found: ID does not exist"
Oct 30 04:14:35.690255 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.690182    4334 scope.go:122] "RemoveContainer" containerID="4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"
Oct 30 04:14:35.690383 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.690367    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38"} err="failed to get container status \"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38\": rpc error: code = NotFound desc = could not find container \"4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38\": container with ID starting with 4dd305b379f8a3afe44565d5290f61985794273bc79a3c07056190b3526c5b38 not found: ID does not exist"
Oct 30 04:14:35.704307 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.704286    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:14:35.715147 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.715124    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:14:35.848896 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.848864    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:35.849117 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:35.849090    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="6fcc8e6d-421d-47de-8659-87fbf1dcbb28" containerName="otc-container" containerID="cri-o://98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552" gracePeriod=30
Oct 30 04:14:36.088120 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.088098    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:36.223439 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.223357    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-tls-certs\") pod \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") "
Oct 30 04:14:36.223439 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.223391    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-data-science-collector-ta-client-cert\") pod \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") "
Oct 30 04:14:36.223439 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.223419    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otc-internal\") pod \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") "
Oct 30 04:14:36.223760 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.223470    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-kube-api-access-d9t6w\" (UniqueName: \"kubernetes.io/projected/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-kube-api-access-d9t6w\") pod \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") "
Oct 30 04:14:36.223760 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.223535    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-monitoring-tls-certs\") pod \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") "
Oct 30 04:14:36.223760 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.223594    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otlp-tls-certs\") pod \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\" (UID: \"6fcc8e6d-421d-47de-8659-87fbf1dcbb28\") "
Oct 30 04:14:36.223937 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.223841    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otc-internal" pod "6fcc8e6d-421d-47de-8659-87fbf1dcbb28" (UID: "6fcc8e6d-421d-47de-8659-87fbf1dcbb28"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:36.225668 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.225643    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otlp-tls-certs" pod "6fcc8e6d-421d-47de-8659-87fbf1dcbb28" (UID: "6fcc8e6d-421d-47de-8659-87fbf1dcbb28"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:36.226118 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.226097    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-tls-certs" pod "6fcc8e6d-421d-47de-8659-87fbf1dcbb28" (UID: "6fcc8e6d-421d-47de-8659-87fbf1dcbb28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:36.226254 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.226230    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-data-science-collector-ta-client-cert" pod "6fcc8e6d-421d-47de-8659-87fbf1dcbb28" (UID: "6fcc8e6d-421d-47de-8659-87fbf1dcbb28"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:36.226311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.226238    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-kube-api-access-d9t6w" pod "6fcc8e6d-421d-47de-8659-87fbf1dcbb28" (UID: "6fcc8e6d-421d-47de-8659-87fbf1dcbb28"). InnerVolumeSpecName "kube-api-access-d9t6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:36.226311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.226269    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-monitoring-tls-certs" pod "6fcc8e6d-421d-47de-8659-87fbf1dcbb28" (UID: "6fcc8e6d-421d-47de-8659-87fbf1dcbb28"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:36.325023 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.324997    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:36.325023 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.325022    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:36.325183 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.325038    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:36.325183 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.325052    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9t6w\" (UniqueName: \"kubernetes.io/projected/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-kube-api-access-d9t6w\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:36.325183 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.325065    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:36.325183 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.325078    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fcc8e6d-421d-47de-8659-87fbf1dcbb28-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:36.662590 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.662555    4334 generic.go:356] "Generic (PLEG): container finished" podID="6fcc8e6d-421d-47de-8659-87fbf1dcbb28" containerID="98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552" exitCode=0
Oct 30 04:14:36.663074 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.662607    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:14:36.663074 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.662641    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6fcc8e6d-421d-47de-8659-87fbf1dcbb28","Type":"ContainerDied","Data":"98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552"}
Oct 30 04:14:36.663074 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.662695    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6fcc8e6d-421d-47de-8659-87fbf1dcbb28","Type":"ContainerDied","Data":"510d13454af3c61df513a6d7c1cd602b84f1121e6a4c4b7dc97989d8fe2acc48"}
Oct 30 04:14:36.663074 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.662717    4334 scope.go:122] "RemoveContainer" containerID="98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552"
Oct 30 04:14:36.671679 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.671660    4334 scope.go:122] "RemoveContainer" containerID="98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552"
Oct 30 04:14:36.671955 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:36.671933    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552\": container with ID starting with 98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552 not found: ID does not exist" containerID="98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552"
Oct 30 04:14:36.672030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.671962    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552"} err="failed to get container status \"98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552\": rpc error: code = NotFound desc = could not find container \"98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552\": container with ID starting with 98eccb99c381cf63346074937aec5d0fa08bdeccd39fa5eed3aa8f290621f552 not found: ID does not exist"
Oct 30 04:14:36.708841 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.708819    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:36.712771 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:36.712749    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:14:37.158642 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:37.158616    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" path="/var/lib/kubelet/pods/2909166b-a7a1-4528-b8c9-a83ba8d7ad67/volumes"
Oct 30 04:14:37.159122 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:37.159107    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fcc8e6d-421d-47de-8659-87fbf1dcbb28" path="/var/lib/kubelet/pods/6fcc8e6d-421d-47de-8659-87fbf1dcbb28/volumes"
Oct 30 04:14:37.247606 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:37.247572    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.117:9093/-/ready\": dial tcp 10.133.0.117:9093: connect: connection refused"
Oct 30 04:14:42.247805 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.247764    4334 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.117:9093/-/ready\": dial tcp 10.133.0.117:9093: connect: connection refused"
Oct 30 04:14:42.913770 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.913713    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:42.914108 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914092    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerName="kube-rbac-proxy"
Oct 30 04:14:42.914108 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914108    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerName="kube-rbac-proxy"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914121    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerName="prom-label-proxy"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914126    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerName="prom-label-proxy"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914136    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="config-reloader"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914143    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="config-reloader"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914156    4334 cpu_manager.go:414] "containerMap: removing container" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerName="kube-rbac-proxy"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914161    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerName="kube-rbac-proxy"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914168    4334 cpu_manager.go:414] "containerMap: removing container" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerName="prom-label-proxy"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914175    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerName="prom-label-proxy"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914190    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="init-config-reloader"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914197    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="init-config-reloader"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914209    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6fcc8e6d-421d-47de-8659-87fbf1dcbb28" containerName="otc-container"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914217    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcc8e6d-421d-47de-8659-87fbf1dcbb28" containerName="otc-container"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914234    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="prometheus"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914241    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="prometheus"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914257    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="thanos-sidecar"
Oct 30 04:14:42.914313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914263    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="thanos-sidecar"
Oct 30 04:14:42.914897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914340    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="prometheus"
Oct 30 04:14:42.914897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914356    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="thanos-sidecar"
Oct 30 04:14:42.914897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914367    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerName="kube-rbac-proxy"
Oct 30 04:14:42.914897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914374    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3b9a83-b96c-439c-9aaa-34895adb3c7a" containerName="prom-label-proxy"
Oct 30 04:14:42.914897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914380    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2909166b-a7a1-4528-b8c9-a83ba8d7ad67" containerName="config-reloader"
Oct 30 04:14:42.914897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914388    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerName="kube-rbac-proxy"
Oct 30 04:14:42.914897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914395    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c1cde-4758-4c4b-8d76-3105831c2f21" containerName="prom-label-proxy"
Oct 30 04:14:42.914897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.914404    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcc8e6d-421d-47de-8659-87fbf1dcbb28" containerName="otc-container"
Oct 30 04:14:42.918931 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.918912    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:42.940766 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.940736    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:14:42.940766 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.940757    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dffss\""
Oct 30 04:14:42.942914 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.942889    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:42.952149 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.952094    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:14:42.952149 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.952131    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:14:42.952319 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.952257    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:14:42.952426 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.952394    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:14:42.975740 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.975711    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9208ba67-d3cc-412d-a318-4d32e36a1b57-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:42.975933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.975764    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:42.975933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.975793    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ck9w\" (UniqueName: \"kubernetes.io/projected/9208ba67-d3cc-412d-a318-4d32e36a1b57-kube-api-access-9ck9w\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:42.975933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.975869    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:42.975933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.975902    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:42.975933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:42.975930    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.077086 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.077055    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9208ba67-d3cc-412d-a318-4d32e36a1b57-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.077248 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.077092    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.077248 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.077117    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ck9w\" (UniqueName: \"kubernetes.io/projected/9208ba67-d3cc-412d-a318-4d32e36a1b57-kube-api-access-9ck9w\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.077248 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.077183    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.077248 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.077216    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.077248 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.077239    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.077970 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:43.077356    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:14:43.077970 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:43.077414    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs podName:9208ba67-d3cc-412d-a318-4d32e36a1b57 nodeName:}" failed. No retries permitted until 2026-10-30 04:14:43.577394659 +0000 UTC m=+5562.583984905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "9208ba67-d3cc-412d-a318-4d32e36a1b57") : secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:14:43.078101 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.078036    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9208ba67-d3cc-412d-a318-4d32e36a1b57-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.079982 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.079958    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.080308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.080290    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.080465 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.080424    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.095685 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.095660    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ck9w\" (UniqueName: \"kubernetes.io/projected/9208ba67-d3cc-412d-a318-4d32e36a1b57-kube-api-access-9ck9w\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.582627 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.582594    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.585162 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.585131    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:43.833870 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:43.833793    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:14:44.125204 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:44.125133    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:14:44.133303 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:14:44.133275    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9208ba67_d3cc_412d_a318_4d32e36a1b57.slice/crio-2115e97b298a34abafbbc3fb83e560d9987d505602341c38b11a70a1dc24b596 WatchSource:0}: Error finding container 2115e97b298a34abafbbc3fb83e560d9987d505602341c38b11a70a1dc24b596: Status 404 returned error can't find the container with id 2115e97b298a34abafbbc3fb83e560d9987d505602341c38b11a70a1dc24b596
Oct 30 04:14:44.135613 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:44.135573    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:14:44.135710 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:44.135652    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:14:44.705201 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:44.705169    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9208ba67-d3cc-412d-a318-4d32e36a1b57","Type":"ContainerStarted","Data":"2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b"}
Oct 30 04:14:44.705632 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:44.705216    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9208ba67-d3cc-412d-a318-4d32e36a1b57","Type":"ContainerStarted","Data":"2115e97b298a34abafbbc3fb83e560d9987d505602341c38b11a70a1dc24b596"}
Oct 30 04:14:45.019201 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.019175    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:45.060375 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.060329    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=3.060316647 podStartE2EDuration="3.060316647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:44.742127342 +0000 UTC m=+5563.748717597" watchObservedRunningTime="2026-10-30 04:14:45.060316647 +0000 UTC m=+5564.066906896"
Oct 30 04:14:45.094084 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.094037    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-config-out\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-config-out\") pod \"23efc980-dc09-4412-b7cc-439358662567\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") "
Oct 30 04:14:45.094084 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.094072    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-tls-assets\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-tls-assets\") pod \"23efc980-dc09-4412-b7cc-439358662567\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") "
Oct 30 04:14:45.094311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.094107    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-web-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-web-config\") pod \"23efc980-dc09-4412-b7cc-439358662567\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") "
Oct 30 04:14:45.094311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.094144    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-kube-api-access-lv5pc\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-kube-api-access-lv5pc\") pod \"23efc980-dc09-4412-b7cc-439358662567\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") "
Oct 30 04:14:45.094311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.094199    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-config-volume\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-config-volume\") pod \"23efc980-dc09-4412-b7cc-439358662567\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") "
Oct 30 04:14:45.094311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.094228    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-cluster-tls-config\") pod \"23efc980-dc09-4412-b7cc-439358662567\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") "
Oct 30 04:14:45.094311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.094262    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-alertmanager-data-science-monitoringstack-db\") pod \"23efc980-dc09-4412-b7cc-439358662567\" (UID: \"23efc980-dc09-4412-b7cc-439358662567\") "
Oct 30 04:14:45.094816 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.094780    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-alertmanager-data-science-monitoringstack-db" pod "23efc980-dc09-4412-b7cc-439358662567" (UID: "23efc980-dc09-4412-b7cc-439358662567"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:14:45.096760 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.096730    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-config-out" pod "23efc980-dc09-4412-b7cc-439358662567" (UID: "23efc980-dc09-4412-b7cc-439358662567"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:14:45.096877 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.096760    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-tls-assets" pod "23efc980-dc09-4412-b7cc-439358662567" (UID: "23efc980-dc09-4412-b7cc-439358662567"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:45.097293 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.097269    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-kube-api-access-lv5pc" pod "23efc980-dc09-4412-b7cc-439358662567" (UID: "23efc980-dc09-4412-b7cc-439358662567"). InnerVolumeSpecName "kube-api-access-lv5pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:45.097750 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.097728    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-config-volume" pod "23efc980-dc09-4412-b7cc-439358662567" (UID: "23efc980-dc09-4412-b7cc-439358662567"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:45.102038 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.102011    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-cluster-tls-config" pod "23efc980-dc09-4412-b7cc-439358662567" (UID: "23efc980-dc09-4412-b7cc-439358662567"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:45.116566 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.116542    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-web-config" pod "23efc980-dc09-4412-b7cc-439358662567" (UID: "23efc980-dc09-4412-b7cc-439358662567"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:45.195805 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.195777    4334 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-config-volume\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:45.195805 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.195807    4334 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-cluster-tls-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:45.195966 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.195817    4334 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:45.195966 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.195827    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23efc980-dc09-4412-b7cc-439358662567-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:45.195966 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.195837    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:45.195966 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.195844    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23efc980-dc09-4412-b7cc-439358662567-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:45.195966 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.195855    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv5pc\" (UniqueName: \"kubernetes.io/projected/23efc980-dc09-4412-b7cc-439358662567-kube-api-access-lv5pc\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:14:45.591621 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.591589    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"]
Oct 30 04:14:45.591912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.591890    4334 cpu_manager.go:414] "containerMap: removing container" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="config-reloader"
Oct 30 04:14:45.591912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.591907    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="config-reloader"
Oct 30 04:14:45.592067 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.591924    4334 cpu_manager.go:414] "containerMap: removing container" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="init-config-reloader"
Oct 30 04:14:45.592067 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.591929    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="init-config-reloader"
Oct 30 04:14:45.592067 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.591939    4334 cpu_manager.go:414] "containerMap: removing container" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="alertmanager"
Oct 30 04:14:45.592067 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.591944    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="alertmanager"
Oct 30 04:14:45.592067 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.592015    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="alertmanager"
Oct 30 04:14:45.592067 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.592026    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="23efc980-dc09-4412-b7cc-439358662567" containerName="config-reloader"
Oct 30 04:14:45.595348 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.595323    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"
Oct 30 04:14:45.609691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.609635    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-fhc4j\""
Oct 30 04:14:45.611329 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.611308    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"]
Oct 30 04:14:45.699642 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.699603    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8lf\" (UniqueName: \"kubernetes.io/projected/0b37f7f4-0ed2-4839-99cc-e91b670eafc7-kube-api-access-2d8lf\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j\" (UID: \"0b37f7f4-0ed2-4839-99cc-e91b670eafc7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"
Oct 30 04:14:45.713759 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.713723    4334 generic.go:356] "Generic (PLEG): container finished" podID="23efc980-dc09-4412-b7cc-439358662567" containerID="7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5" exitCode=0
Oct 30 04:14:45.714083 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.713789    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:45.714083 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.713808    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"23efc980-dc09-4412-b7cc-439358662567","Type":"ContainerDied","Data":"7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5"}
Oct 30 04:14:45.714083 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.713897    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"23efc980-dc09-4412-b7cc-439358662567","Type":"ContainerDied","Data":"4794e404305d7e28921c9dfdee927d601c386e9d5d4e3afd7b6c899c11c4c1bc"}
Oct 30 04:14:45.714083 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.713926    4334 scope.go:122] "RemoveContainer" containerID="174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086"
Oct 30 04:14:45.723004 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.722988    4334 scope.go:122] "RemoveContainer" containerID="7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5"
Oct 30 04:14:45.730493 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.730468    4334 scope.go:122] "RemoveContainer" containerID="3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d"
Oct 30 04:14:45.737784 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.737757    4334 scope.go:122] "RemoveContainer" containerID="174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086"
Oct 30 04:14:45.738025 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:45.738000    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086\": container with ID starting with 174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086 not found: ID does not exist" containerID="174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086"
Oct 30 04:14:45.738089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.738025    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086"} err="failed to get container status \"174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086\": rpc error: code = NotFound desc = could not find container \"174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086\": container with ID starting with 174578e10ff449f50168eae306b1da287931882730610ff50238e202712ca086 not found: ID does not exist"
Oct 30 04:14:45.738089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.738042    4334 scope.go:122] "RemoveContainer" containerID="7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5"
Oct 30 04:14:45.738239 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:45.738224    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5\": container with ID starting with 7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5 not found: ID does not exist" containerID="7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5"
Oct 30 04:14:45.738281 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.738243    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5"} err="failed to get container status \"7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5\": rpc error: code = NotFound desc = could not find container \"7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5\": container with ID starting with 7d6663692f05b7e1ec9a7a013cb397da8672021d36fd8d195fd1859ba93aa3b5 not found: ID does not exist"
Oct 30 04:14:45.738281 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.738254    4334 scope.go:122] "RemoveContainer" containerID="3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d"
Oct 30 04:14:45.738441 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:45.738425    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d\": container with ID starting with 3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d not found: ID does not exist" containerID="3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d"
Oct 30 04:14:45.738514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.738460    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d"} err="failed to get container status \"3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d\": rpc error: code = NotFound desc = could not find container \"3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d\": container with ID starting with 3ca30d8dd1c8ca56a8a6694ccf4aa3596ea8500a562baa633fd6a833df949c8d not found: ID does not exist"
Oct 30 04:14:45.761291 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.761269    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:14:45.772249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.772226    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:14:45.800177 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.800150    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8lf\" (UniqueName: \"kubernetes.io/projected/0b37f7f4-0ed2-4839-99cc-e91b670eafc7-kube-api-access-2d8lf\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j\" (UID: \"0b37f7f4-0ed2-4839-99cc-e91b670eafc7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"
Oct 30 04:14:45.818397 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.818378    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8lf\" (UniqueName: \"kubernetes.io/projected/0b37f7f4-0ed2-4839-99cc-e91b670eafc7-kube-api-access-2d8lf\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j\" (UID: \"0b37f7f4-0ed2-4839-99cc-e91b670eafc7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"
Oct 30 04:14:45.907081 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:45.907061    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"
Oct 30 04:14:46.204314 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.204243    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"]
Oct 30 04:14:46.208570 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:14:46.208542    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b37f7f4_0ed2_4839_99cc_e91b670eafc7.slice/crio-757fabed0dd1293335bfb54d715465968ae33f822cf07aa9fa247be2012a82ee WatchSource:0}: Error finding container 757fabed0dd1293335bfb54d715465968ae33f822cf07aa9fa247be2012a82ee: Status 404 returned error can't find the container with id 757fabed0dd1293335bfb54d715465968ae33f822cf07aa9fa247be2012a82ee
Oct 30 04:14:46.403284 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.403228    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:14:46.407278 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.407257    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.428005 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.427975    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:14:46.498506 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.498366    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:14:46.498506 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.498411    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:14:46.498506 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.498434    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:14:46.498506 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.498468    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:14:46.498741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.498514    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-2brw5\""
Oct 30 04:14:46.503904 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.503882    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.503951 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.503913    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.503951 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.503933    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wp4j\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-kube-api-access-9wp4j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.504092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.504075    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.504127 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.504116    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.504172 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.504161    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.504209 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.504187    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.542668 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.542641    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:14:46.546435 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.546417    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.560494 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.560471    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:14:46.560633 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.560589    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:14:46.560888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.560870    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:14:46.571145 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.571123    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:14:46.602615 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.602576    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:14:46.602723 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.602641    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:14:46.602723 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.602642    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:14:46.602929 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.602912    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:14:46.602929 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.602923    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:14:46.603029 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.602944    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:14:46.603029 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.602985    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-9zb5b\""
Oct 30 04:14:46.605235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605212    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.605318 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605252    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4526ca2d-c26f-4d26-b646-83edefe10f53-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.605318 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605284    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.605435 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605419    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.605498 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605479    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.605533 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605508    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.605624 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605598    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.605744 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605686    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.605744 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605712    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.605857 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605836    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wp4j\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-kube-api-access-9wp4j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.605916 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605888    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.605976 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605918    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.605976 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605954    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.606078 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.605984    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.606078 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.606013    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bmw\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-kube-api-access-24bmw\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.606078 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.606042    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.606078 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.606070    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.606280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.606094    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.606280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.606120    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.606669 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.606640    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.607925 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.607882    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.608532 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.608227    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.608651 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.608544    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.609051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.609029    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.610081 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.610056    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.623159 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.623137    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wp4j\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-kube-api-access-9wp4j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.707285 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707260    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707309    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707329    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707358    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707376    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707392    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bmw\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-kube-api-access-24bmw\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707413    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707438    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707502    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4526ca2d-c26f-4d26-b646-83edefe10f53-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707529    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707571    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.707733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.707596    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.709015 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.708126    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.709015 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.708439    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.709015 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.708701    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.710235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.710212    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4526ca2d-c26f-4d26-b646-83edefe10f53-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.710338 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.710290    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.710489 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.710443    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.710577 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.710487    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.710577 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.710520    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.710985 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.710966    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.711273 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.711255    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.711320 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.711278    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.720309 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.720281    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j" event={"ID":"0b37f7f4-0ed2-4839-99cc-e91b670eafc7","Type":"ContainerStarted","Data":"c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b"}
Oct 30 04:14:46.720681 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.720325    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j" event={"ID":"0b37f7f4-0ed2-4839-99cc-e91b670eafc7","Type":"ContainerStarted","Data":"757fabed0dd1293335bfb54d715465968ae33f822cf07aa9fa247be2012a82ee"}
Oct 30 04:14:46.724463 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.724434    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:46.725038 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.725015    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bmw\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-kube-api-access-24bmw\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:46.755013 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:46.754935    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j" podStartSLOduration=1.7549040599999999 podStartE2EDuration="1.75490406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:46.753598471 +0000 UTC m=+5565.760188725" watchObservedRunningTime="2026-10-30 04:14:46.75490406 +0000 UTC m=+5565.761494315"
Oct 30 04:14:47.040902 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:47.040819    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:14:47.044602 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:14:47.044574    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a25202_a0a3_4517_a73a_311babf85515.slice/crio-3275bd62912c9049dfb3b50415d2b9fe166ab9d6a3e80cbb9e4541dfd10af3be WatchSource:0}: Error finding container 3275bd62912c9049dfb3b50415d2b9fe166ab9d6a3e80cbb9e4541dfd10af3be: Status 404 returned error can't find the container with id 3275bd62912c9049dfb3b50415d2b9fe166ab9d6a3e80cbb9e4541dfd10af3be
Oct 30 04:14:47.158643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:47.158615    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23efc980-dc09-4412-b7cc-439358662567" path="/var/lib/kubelet/pods/23efc980-dc09-4412-b7cc-439358662567/volumes"
Oct 30 04:14:47.728079 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:47.728040    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"e2a25202-a0a3-4517-a73a-311babf85515","Type":"ContainerStarted","Data":"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3"}
Oct 30 04:14:47.728515 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:47.728104    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"e2a25202-a0a3-4517-a73a-311babf85515","Type":"ContainerStarted","Data":"3275bd62912c9049dfb3b50415d2b9fe166ab9d6a3e80cbb9e4541dfd10af3be"}
Oct 30 04:14:49.629920 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:49.629884    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:49.655924 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:49.655898    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:49.731226 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:49.731199    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:49.731349 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:49.731320    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:49.744232 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:49.744207    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") DevicePath \"csi-e364dd4f86bda642cef8d14035ca0bfc3cdafe8ca20f42d905720eb71a97db6d\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:49.841795 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:49.841754    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8c516923ea0339579ebf808cef81b9e0aaf55e5b619e2c6187025fa3b2b51f7f/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:49.867925 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:49.867900    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:50.165720 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:50.165686    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:14:50.494583 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:14:50.494552    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4526ca2d_c26f_4d26_b646_83edefe10f53.slice/crio-d34ed9bc0ffcadac6274aed69678872fbeed8901b16c0c84c999ed9e17ff3845 WatchSource:0}: Error finding container d34ed9bc0ffcadac6274aed69678872fbeed8901b16c0c84c999ed9e17ff3845: Status 404 returned error can't find the container with id d34ed9bc0ffcadac6274aed69678872fbeed8901b16c0c84c999ed9e17ff3845
Oct 30 04:14:50.502611 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:50.502590    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:14:50.742974 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:50.742904    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerStarted","Data":"d34ed9bc0ffcadac6274aed69678872fbeed8901b16c0c84c999ed9e17ff3845"}
Oct 30 04:14:50.978673 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:14:50.978638    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:14:51.750148 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:51.750112    4334 generic.go:356] "Generic (PLEG): container finished" podID="e2a25202-a0a3-4517-a73a-311babf85515" containerID="91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3" exitCode=0
Oct 30 04:14:51.750556 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:51.750182    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"e2a25202-a0a3-4517-a73a-311babf85515","Type":"ContainerDied","Data":"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3"}
Oct 30 04:14:51.751915 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:51.751891    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerStarted","Data":"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"}
Oct 30 04:14:52.758583 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:52.758552    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"e2a25202-a0a3-4517-a73a-311babf85515","Type":"ContainerStarted","Data":"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821"}
Oct 30 04:14:52.759048 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:52.758612    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"e2a25202-a0a3-4517-a73a-311babf85515","Type":"ContainerStarted","Data":"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd"}
Oct 30 04:14:52.759048 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:52.758911    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:14:52.799087 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:52.799043    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.412467429 podStartE2EDuration="6.799031847s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.386564418s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:52.797616355 +0000 UTC m=+5571.804206591" watchObservedRunningTime="2026-10-30 04:14:52.799031847 +0000 UTC m=+5571.805622151"
Oct 30 04:14:54.770225 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:54.770193    4334 generic.go:356] "Generic (PLEG): container finished" podID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerID="b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e" exitCode=0
Oct 30 04:14:54.770608 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:54.770263    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerDied","Data":"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"}
Oct 30 04:14:55.776939 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:55.776905    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerStarted","Data":"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"}
Oct 30 04:14:55.777407 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:55.776979    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerStarted","Data":"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"}
Oct 30 04:14:55.777407 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:55.777020    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerStarted","Data":"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"}
Oct 30 04:14:55.836545 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:55.836428    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=6.34880295 podStartE2EDuration="9.836415455s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.487612505s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:55.825075152 +0000 UTC m=+5574.831665419" watchObservedRunningTime="2026-10-30 04:14:55.836415455 +0000 UTC m=+5574.843005762"
Oct 30 04:14:58.765696 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:14:58.765658    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:00.166666 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:00.166636    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:05.163787 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.163759    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:05.164241 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.163961    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="9208ba67-d3cc-412d-a318-4d32e36a1b57" containerName="otc-container" containerID="cri-o://2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b" gracePeriod=30
Oct 30 04:15:05.166527 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.166492    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:05.197919 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.197900    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:05.404949 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.404926    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:05.556484 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.556384    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9208ba67-d3cc-412d-a318-4d32e36a1b57-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9208ba67-d3cc-412d-a318-4d32e36a1b57-otc-internal\") pod \"9208ba67-d3cc-412d-a318-4d32e36a1b57\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") "
Oct 30 04:15:05.556659 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.556483    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-data-science-collector-ta-client-cert\") pod \"9208ba67-d3cc-412d-a318-4d32e36a1b57\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") "
Oct 30 04:15:05.556659 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.556526    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-tls-certs\") pod \"9208ba67-d3cc-412d-a318-4d32e36a1b57\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") "
Oct 30 04:15:05.556659 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.556573    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-monitoring-tls-certs\") pod \"9208ba67-d3cc-412d-a318-4d32e36a1b57\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") "
Oct 30 04:15:05.556659 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.556599    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs\") pod \"9208ba67-d3cc-412d-a318-4d32e36a1b57\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") "
Oct 30 04:15:05.556659 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.556647    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9208ba67-d3cc-412d-a318-4d32e36a1b57-kube-api-access-9ck9w\" (UniqueName: \"kubernetes.io/projected/9208ba67-d3cc-412d-a318-4d32e36a1b57-kube-api-access-9ck9w\") pod \"9208ba67-d3cc-412d-a318-4d32e36a1b57\" (UID: \"9208ba67-d3cc-412d-a318-4d32e36a1b57\") "
Oct 30 04:15:05.556938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.556830    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9208ba67-d3cc-412d-a318-4d32e36a1b57-otc-internal" pod "9208ba67-d3cc-412d-a318-4d32e36a1b57" (UID: "9208ba67-d3cc-412d-a318-4d32e36a1b57"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:05.557002 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.556939    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9208ba67-d3cc-412d-a318-4d32e36a1b57-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:05.558790 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.558757    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-data-science-collector-ta-client-cert" pod "9208ba67-d3cc-412d-a318-4d32e36a1b57" (UID: "9208ba67-d3cc-412d-a318-4d32e36a1b57"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:05.558912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.558819    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-tls-certs" pod "9208ba67-d3cc-412d-a318-4d32e36a1b57" (UID: "9208ba67-d3cc-412d-a318-4d32e36a1b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:05.558912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.558872    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs" pod "9208ba67-d3cc-412d-a318-4d32e36a1b57" (UID: "9208ba67-d3cc-412d-a318-4d32e36a1b57"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:05.559097 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.559082    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-monitoring-tls-certs" pod "9208ba67-d3cc-412d-a318-4d32e36a1b57" (UID: "9208ba67-d3cc-412d-a318-4d32e36a1b57"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:05.559185 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.559161    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9208ba67-d3cc-412d-a318-4d32e36a1b57-kube-api-access-9ck9w" pod "9208ba67-d3cc-412d-a318-4d32e36a1b57" (UID: "9208ba67-d3cc-412d-a318-4d32e36a1b57"). InnerVolumeSpecName "kube-api-access-9ck9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:05.657723 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.657693    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:05.657723 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.657718    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ck9w\" (UniqueName: \"kubernetes.io/projected/9208ba67-d3cc-412d-a318-4d32e36a1b57-kube-api-access-9ck9w\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:05.657723 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.657729    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:05.657932 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.657739    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:05.657932 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.657748    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9208ba67-d3cc-412d-a318-4d32e36a1b57-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:05.827722 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.827653    4334 generic.go:356] "Generic (PLEG): container finished" podID="9208ba67-d3cc-412d-a318-4d32e36a1b57" containerID="2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b" exitCode=0
Oct 30 04:15:05.827722 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.827697    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:05.827891 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.827741    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9208ba67-d3cc-412d-a318-4d32e36a1b57","Type":"ContainerDied","Data":"2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b"}
Oct 30 04:15:05.827891 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.827792    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9208ba67-d3cc-412d-a318-4d32e36a1b57","Type":"ContainerDied","Data":"2115e97b298a34abafbbc3fb83e560d9987d505602341c38b11a70a1dc24b596"}
Oct 30 04:15:05.827891 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.827814    4334 scope.go:122] "RemoveContainer" containerID="2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b"
Oct 30 04:15:05.832766 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.832744    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:05.837311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.837296    4334 scope.go:122] "RemoveContainer" containerID="2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b"
Oct 30 04:15:05.837600 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:05.837584    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b\": container with ID starting with 2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b not found: ID does not exist" containerID="2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b"
Oct 30 04:15:05.837656 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.837610    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b"} err="failed to get container status \"2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b\": rpc error: code = NotFound desc = could not find container \"2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b\": container with ID starting with 2184cc4fc388035365defd1d4c6ccc6c8093983fe2a5376e1aba4def6cf22e3b not found: ID does not exist"
Oct 30 04:15:05.921413 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.921382    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:05.925434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.925408    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:05.963148 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.963121    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:05.963507 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.963492    4334 cpu_manager.go:414] "containerMap: removing container" podUID="9208ba67-d3cc-412d-a318-4d32e36a1b57" containerName="otc-container"
Oct 30 04:15:05.963507 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.963507    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9208ba67-d3cc-412d-a318-4d32e36a1b57" containerName="otc-container"
Oct 30 04:15:05.963610 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.963560    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="9208ba67-d3cc-412d-a318-4d32e36a1b57" containerName="otc-container"
Oct 30 04:15:05.967763 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.967740    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:05.982045 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.982025    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:15:05.982284 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.982256    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:15:05.982385 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.982365    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dffss\""
Oct 30 04:15:05.982385 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.982373    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:15:05.982502 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.982428    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:15:05.982502 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.982428    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:15:05.989271 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:05.989251    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:06.061092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.061064    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.061235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.061112    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.061235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.061186    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk6q9\" (UniqueName: \"kubernetes.io/projected/61418f8d-0997-433f-ad5a-01b05f796244-kube-api-access-bk6q9\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.061235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.061207    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.061362 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.061261    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/61418f8d-0997-433f-ad5a-01b05f796244-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.061362 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.061326    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.162829 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.162788    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.162999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.162868    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk6q9\" (UniqueName: \"kubernetes.io/projected/61418f8d-0997-433f-ad5a-01b05f796244-kube-api-access-bk6q9\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.162999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.162900    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.162999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.162928    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/61418f8d-0997-433f-ad5a-01b05f796244-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.163161 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.163017    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.163161 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.163145    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.163765 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.163739    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/61418f8d-0997-433f-ad5a-01b05f796244-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.165720 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.165699    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.166046 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.165760    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.166046 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.165820    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.166046 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.165843    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.180790 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.180762    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk6q9\" (UniqueName: \"kubernetes.io/projected/61418f8d-0997-433f-ad5a-01b05f796244-kube-api-access-bk6q9\") pod \"data-science-collector-collector-0\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.280070 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.280045    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:06.574427 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.574338    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:06.578051 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:06.578022    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61418f8d_0997_433f_ad5a_01b05f796244.slice/crio-0cd721afa6aa5892008735549f9fc0641c5890188a66682994119906de1d9d88 WatchSource:0}: Error finding container 0cd721afa6aa5892008735549f9fc0641c5890188a66682994119906de1d9d88: Status 404 returned error can't find the container with id 0cd721afa6aa5892008735549f9fc0641c5890188a66682994119906de1d9d88
Oct 30 04:15:06.580321 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.580285    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:15:06.580394 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.580350    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:15:06.834229 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.834142    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"61418f8d-0997-433f-ad5a-01b05f796244","Type":"ContainerStarted","Data":"1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d"}
Oct 30 04:15:06.834229 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.834196    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"61418f8d-0997-433f-ad5a-01b05f796244","Type":"ContainerStarted","Data":"0cd721afa6aa5892008735549f9fc0641c5890188a66682994119906de1d9d88"}
Oct 30 04:15:06.889064 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:06.889018    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.889004523 podStartE2EDuration="1.889004523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:06.888491442 +0000 UTC m=+5585.895081697" watchObservedRunningTime="2026-10-30 04:15:06.889004523 +0000 UTC m=+5585.895594777"
Oct 30 04:15:07.159897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:07.159860    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9208ba67-d3cc-412d-a318-4d32e36a1b57" path="/var/lib/kubelet/pods/9208ba67-d3cc-412d-a318-4d32e36a1b57/volumes"
Oct 30 04:15:16.215938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:16.215903    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:16.216769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:16.216383    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="61418f8d-0997-433f-ad5a-01b05f796244" containerName="otc-container" containerID="cri-o://1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d" gracePeriod=30
Oct 30 04:15:17.560821 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.560798    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:17.661810 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.661779    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-data-science-collector-ta-client-cert\") pod \"61418f8d-0997-433f-ad5a-01b05f796244\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") "
Oct 30 04:15:17.661992 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.661870    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61418f8d-0997-433f-ad5a-01b05f796244-otc-internal\" (UniqueName: \"kubernetes.io/configmap/61418f8d-0997-433f-ad5a-01b05f796244-otc-internal\") pod \"61418f8d-0997-433f-ad5a-01b05f796244\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") "
Oct 30 04:15:17.661992 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.661905    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-monitoring-tls-certs\") pod \"61418f8d-0997-433f-ad5a-01b05f796244\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") "
Oct 30 04:15:17.661992 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.661944    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-tls-certs\") pod \"61418f8d-0997-433f-ad5a-01b05f796244\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") "
Oct 30 04:15:17.662166 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.661993    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-otlp-tls-certs\") pod \"61418f8d-0997-433f-ad5a-01b05f796244\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") "
Oct 30 04:15:17.662166 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.662022    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61418f8d-0997-433f-ad5a-01b05f796244-kube-api-access-bk6q9\" (UniqueName: \"kubernetes.io/projected/61418f8d-0997-433f-ad5a-01b05f796244-kube-api-access-bk6q9\") pod \"61418f8d-0997-433f-ad5a-01b05f796244\" (UID: \"61418f8d-0997-433f-ad5a-01b05f796244\") "
Oct 30 04:15:17.662346 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.662318    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61418f8d-0997-433f-ad5a-01b05f796244-otc-internal" pod "61418f8d-0997-433f-ad5a-01b05f796244" (UID: "61418f8d-0997-433f-ad5a-01b05f796244"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:17.664241 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.664206    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61418f8d-0997-433f-ad5a-01b05f796244-kube-api-access-bk6q9" pod "61418f8d-0997-433f-ad5a-01b05f796244" (UID: "61418f8d-0997-433f-ad5a-01b05f796244"). InnerVolumeSpecName "kube-api-access-bk6q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:17.664241 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.664230    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-otlp-tls-certs" pod "61418f8d-0997-433f-ad5a-01b05f796244" (UID: "61418f8d-0997-433f-ad5a-01b05f796244"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:17.664428 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.664216    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-tls-certs" pod "61418f8d-0997-433f-ad5a-01b05f796244" (UID: "61418f8d-0997-433f-ad5a-01b05f796244"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:17.664515 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.664505    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-monitoring-tls-certs" pod "61418f8d-0997-433f-ad5a-01b05f796244" (UID: "61418f8d-0997-433f-ad5a-01b05f796244"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:17.664579 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.664521    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-data-science-collector-ta-client-cert" pod "61418f8d-0997-433f-ad5a-01b05f796244" (UID: "61418f8d-0997-433f-ad5a-01b05f796244"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:17.763343 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.763316    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/61418f8d-0997-433f-ad5a-01b05f796244-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:17.763343 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.763341    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:17.763541 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.763351    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:17.763541 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.763359    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:17.763541 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.763367    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk6q9\" (UniqueName: \"kubernetes.io/projected/61418f8d-0997-433f-ad5a-01b05f796244-kube-api-access-bk6q9\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:17.763541 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.763379    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/61418f8d-0997-433f-ad5a-01b05f796244-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:17.890505 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.890438    4334 generic.go:356] "Generic (PLEG): container finished" podID="61418f8d-0997-433f-ad5a-01b05f796244" containerID="1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d" exitCode=0
Oct 30 04:15:17.890668 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.890507    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:17.890668 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.890547    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"61418f8d-0997-433f-ad5a-01b05f796244","Type":"ContainerDied","Data":"1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d"}
Oct 30 04:15:17.890668 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.890604    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"61418f8d-0997-433f-ad5a-01b05f796244","Type":"ContainerDied","Data":"0cd721afa6aa5892008735549f9fc0641c5890188a66682994119906de1d9d88"}
Oct 30 04:15:17.890668 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.890632    4334 scope.go:122] "RemoveContainer" containerID="1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d"
Oct 30 04:15:17.899313 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.899297    4334 scope.go:122] "RemoveContainer" containerID="1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d"
Oct 30 04:15:17.899699 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:17.899680    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d\": container with ID starting with 1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d not found: ID does not exist" containerID="1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d"
Oct 30 04:15:17.899766 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.899709    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d"} err="failed to get container status \"1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d\": rpc error: code = NotFound desc = could not find container \"1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d\": container with ID starting with 1a037785422abfe56860951990d82b4f3effc1734e415155ed63e36ebea6d86d not found: ID does not exist"
Oct 30 04:15:17.937751 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.937697    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:17.941296 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.941275    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:17.977004 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.976979    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:17.977329 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.977315    4334 cpu_manager.go:414] "containerMap: removing container" podUID="61418f8d-0997-433f-ad5a-01b05f796244" containerName="otc-container"
Oct 30 04:15:17.977371 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.977331    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61418f8d-0997-433f-ad5a-01b05f796244" containerName="otc-container"
Oct 30 04:15:17.977410 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.977402    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="61418f8d-0997-433f-ad5a-01b05f796244" containerName="otc-container"
Oct 30 04:15:17.981661 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.981641    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:17.997323 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.997286    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:15:17.997583 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.997534    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dffss\""
Oct 30 04:15:17.997710 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.997685    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:15:17.997827 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.997763    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:15:17.998030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.998011    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:15:17.998030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.998018    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:15:17.999544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:17.999523    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:18.065734 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.065701    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.065882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.065764    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.065882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.065838    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.065962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.065891    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.065962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.065927    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.065962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.065950    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/d664440a-1f61-4b27-9f21-53d08f8b37c0-kube-api-access-d97wh\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.167066 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.167036    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.167190 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.167087    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.167254 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.167197    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.167300 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.167249    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.167300 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.167282    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.167404 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.167319    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/d664440a-1f61-4b27-9f21-53d08f8b37c0-kube-api-access-d97wh\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.167799 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.167776    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.169618 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.169598    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.169773 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.169737    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.170010 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.169991    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.170095 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.170074    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.185052 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.185025    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/d664440a-1f61-4b27-9f21-53d08f8b37c0-kube-api-access-d97wh\") pod \"data-science-collector-collector-0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.295303 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.295240    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:18.588908 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.588824    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:18.592942 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:18.592910    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd664440a_1f61_4b27_9f21_53d08f8b37c0.slice/crio-71b8f7eb0881cf41ff4e356218a7bf1a8011df38deb50499d367bea35a3a9c89 WatchSource:0}: Error finding container 71b8f7eb0881cf41ff4e356218a7bf1a8011df38deb50499d367bea35a3a9c89: Status 404 returned error can't find the container with id 71b8f7eb0881cf41ff4e356218a7bf1a8011df38deb50499d367bea35a3a9c89
Oct 30 04:15:18.595053 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.595015    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:15:18.595156 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.595087    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:15:18.896693 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.896031    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"]
Oct 30 04:15:18.896693 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.896270    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j" podUID="0b37f7f4-0ed2-4839-99cc-e91b670eafc7" containerName="thanos-querier" containerID="cri-o://c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b" gracePeriod=30
Oct 30 04:15:18.902060 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.902027    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d664440a-1f61-4b27-9f21-53d08f8b37c0","Type":"ContainerStarted","Data":"fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662"}
Oct 30 04:15:18.902213 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.902088    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d664440a-1f61-4b27-9f21-53d08f8b37c0","Type":"ContainerStarted","Data":"71b8f7eb0881cf41ff4e356218a7bf1a8011df38deb50499d367bea35a3a9c89"}
Oct 30 04:15:18.988030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:18.984028    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.9840077329999999 podStartE2EDuration="1.984007733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:18.982994939 +0000 UTC m=+5597.989585219" watchObservedRunningTime="2026-10-30 04:15:18.984007733 +0000 UTC m=+5597.990597989"
Oct 30 04:15:19.037813 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.037781    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:15:19.038178 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.038105    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="alertmanager" containerID="cri-o://87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd" gracePeriod=120
Oct 30 04:15:19.038294 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.038164    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="config-reloader" containerID="cri-o://336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821" gracePeriod=120
Oct 30 04:15:19.053677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.053647    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:15:19.054319 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.054286    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="prometheus" containerID="cri-o://31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec" gracePeriod=600
Oct 30 04:15:19.055154 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.055126    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="thanos-sidecar" containerID="cri-o://0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5" gracePeriod=600
Oct 30 04:15:19.055930 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.055318    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="config-reloader" containerID="cri-o://a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11" gracePeriod=600
Oct 30 04:15:19.158590 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.158557    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61418f8d-0997-433f-ad5a-01b05f796244" path="/var/lib/kubelet/pods/61418f8d-0997-433f-ad5a-01b05f796244/volumes"
Oct 30 04:15:19.171762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.171738    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"
Oct 30 04:15:19.291859 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.291833    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b37f7f4-0ed2-4839-99cc-e91b670eafc7-kube-api-access-2d8lf\" (UniqueName: \"kubernetes.io/projected/0b37f7f4-0ed2-4839-99cc-e91b670eafc7-kube-api-access-2d8lf\") pod \"0b37f7f4-0ed2-4839-99cc-e91b670eafc7\" (UID: \"0b37f7f4-0ed2-4839-99cc-e91b670eafc7\") "
Oct 30 04:15:19.293845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.293824    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b37f7f4-0ed2-4839-99cc-e91b670eafc7-kube-api-access-2d8lf" pod "0b37f7f4-0ed2-4839-99cc-e91b670eafc7" (UID: "0b37f7f4-0ed2-4839-99cc-e91b670eafc7"). InnerVolumeSpecName "kube-api-access-2d8lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:19.392676 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.392645    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8lf\" (UniqueName: \"kubernetes.io/projected/0b37f7f4-0ed2-4839-99cc-e91b670eafc7-kube-api-access-2d8lf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:19.652578 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.652486    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:19.838294 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.838268    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:19.898170 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898143    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.898310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898176    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-config\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.898310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898199    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-tls-assets\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-tls-assets\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.898310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898234    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.898310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898268    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.898310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898301    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-thanos-prometheus-http-client-file\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.902670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898399    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.902670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898497    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4526ca2d-c26f-4d26-b646-83edefe10f53-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4526ca2d-c26f-4d26-b646-83edefe10f53-config-out\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.902670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898542    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.902670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.898575    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-kube-api-access-24bmw\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-kube-api-access-24bmw\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.902670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.899056    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.902670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.899154    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.902670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.899209    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-secret-prometheus-web-tls-ca\") pod \"4526ca2d-c26f-4d26-b646-83edefe10f53\" (UID: \"4526ca2d-c26f-4d26-b646-83edefe10f53\") "
Oct 30 04:15:19.902670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.899613    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-1" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:19.903476 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.903428    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-2" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:19.903725 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.903697    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-0" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:19.906947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.906924    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-secret-prometheus-web-tls-ca" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.907670 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.907643    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4526ca2d-c26f-4d26-b646-83edefe10f53-config-out" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:19.907784 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.907773    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-thanos-prometheus-http-client-file" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.908083 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.908049    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.908423 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.908396    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-tls-assets" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:19.908897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.908678    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-config" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.910351 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.909801    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "pvc-9689c530-7f91-4584-b28e-8809bed9e799". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:15:19.910351 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.910026    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.911297 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.911264    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-kube-api-access-24bmw" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "kube-api-access-24bmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:19.911815 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.911789    4334 generic.go:356] "Generic (PLEG): container finished" podID="0b37f7f4-0ed2-4839-99cc-e91b670eafc7" containerID="c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b" exitCode=0
Oct 30 04:15:19.911889 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.911875    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j" event={"ID":"0b37f7f4-0ed2-4839-99cc-e91b670eafc7","Type":"ContainerDied","Data":"c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b"}
Oct 30 04:15:19.911933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.911910    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j" event={"ID":"0b37f7f4-0ed2-4839-99cc-e91b670eafc7","Type":"ContainerDied","Data":"757fabed0dd1293335bfb54d715465968ae33f822cf07aa9fa247be2012a82ee"}
Oct 30 04:15:19.911968 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.911932    4334 scope.go:122] "RemoveContainer" containerID="c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b"
Oct 30 04:15:19.912057 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.912045    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"
Oct 30 04:15:19.913726 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.913705    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:19.916310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.916286    4334 generic.go:356] "Generic (PLEG): container finished" podID="e2a25202-a0a3-4517-a73a-311babf85515" containerID="336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821" exitCode=0
Oct 30 04:15:19.916416 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.916311    4334 generic.go:356] "Generic (PLEG): container finished" podID="e2a25202-a0a3-4517-a73a-311babf85515" containerID="87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd" exitCode=0
Oct 30 04:15:19.916416 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.916372    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"e2a25202-a0a3-4517-a73a-311babf85515","Type":"ContainerDied","Data":"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821"}
Oct 30 04:15:19.916580 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.916554    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"e2a25202-a0a3-4517-a73a-311babf85515","Type":"ContainerDied","Data":"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd"}
Oct 30 04:15:19.916653 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.916603    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"e2a25202-a0a3-4517-a73a-311babf85515","Type":"ContainerDied","Data":"3275bd62912c9049dfb3b50415d2b9fe166ab9d6a3e80cbb9e4541dfd10af3be"}
Oct 30 04:15:19.920075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.919720    4334 generic.go:356] "Generic (PLEG): container finished" podID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerID="0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5" exitCode=0
Oct 30 04:15:19.920075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.919742    4334 generic.go:356] "Generic (PLEG): container finished" podID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerID="a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11" exitCode=0
Oct 30 04:15:19.920075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.919752    4334 generic.go:356] "Generic (PLEG): container finished" podID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerID="31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec" exitCode=0
Oct 30 04:15:19.920075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.919803    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:15:19.920075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.919893    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerDied","Data":"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"}
Oct 30 04:15:19.920075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.919947    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerDied","Data":"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"}
Oct 30 04:15:19.920075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.919985    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerDied","Data":"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"}
Oct 30 04:15:19.920075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.920022    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4526ca2d-c26f-4d26-b646-83edefe10f53","Type":"ContainerDied","Data":"d34ed9bc0ffcadac6274aed69678872fbeed8901b16c0c84c999ed9e17ff3845"}
Oct 30 04:15:19.920409 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.920181    4334 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-dffss\" not found"
Oct 30 04:15:19.922481 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.922434    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config" pod "4526ca2d-c26f-4d26-b646-83edefe10f53" (UID: "4526ca2d-c26f-4d26-b646-83edefe10f53"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:19.925270 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.925252    4334 scope.go:122] "RemoveContainer" containerID="c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b"
Oct 30 04:15:19.925663 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:19.925646    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b\": container with ID starting with c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b not found: ID does not exist" containerID="c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b"
Oct 30 04:15:19.925709 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.925669    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b"} err="failed to get container status \"c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b\": rpc error: code = NotFound desc = could not find container \"c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b\": container with ID starting with c0bc6420ea7513d9555479bc9f3d4bffd655758180e35042edf701c8b7e0600b not found: ID does not exist"
Oct 30 04:15:19.925709 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.925684    4334 scope.go:122] "RemoveContainer" containerID="336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821"
Oct 30 04:15:19.933101 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.933086    4334 scope.go:122] "RemoveContainer" containerID="87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd"
Oct 30 04:15:19.942034 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.942011    4334 scope.go:122] "RemoveContainer" containerID="91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3"
Oct 30 04:15:19.949346 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.949318    4334 scope.go:122] "RemoveContainer" containerID="336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821"
Oct 30 04:15:19.949638 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:19.949618    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821\": container with ID starting with 336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821 not found: ID does not exist" containerID="336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821"
Oct 30 04:15:19.949697 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.949646    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821"} err="failed to get container status \"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821\": rpc error: code = NotFound desc = could not find container \"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821\": container with ID starting with 336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821 not found: ID does not exist"
Oct 30 04:15:19.949697 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.949662    4334 scope.go:122] "RemoveContainer" containerID="87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd"
Oct 30 04:15:19.949917 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:19.949898    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd\": container with ID starting with 87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd not found: ID does not exist" containerID="87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd"
Oct 30 04:15:19.949959 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.949923    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd"} err="failed to get container status \"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd\": rpc error: code = NotFound desc = could not find container \"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd\": container with ID starting with 87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd not found: ID does not exist"
Oct 30 04:15:19.949959 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.949938    4334 scope.go:122] "RemoveContainer" containerID="91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3"
Oct 30 04:15:19.950192 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:19.950167    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3\": container with ID starting with 91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3 not found: ID does not exist" containerID="91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3"
Oct 30 04:15:19.950261 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.950203    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3"} err="failed to get container status \"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3\": rpc error: code = NotFound desc = could not find container \"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3\": container with ID starting with 91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3 not found: ID does not exist"
Oct 30 04:15:19.950261 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.950227    4334 scope.go:122] "RemoveContainer" containerID="336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821"
Oct 30 04:15:19.950516 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.950488    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821"} err="failed to get container status \"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821\": rpc error: code = NotFound desc = could not find container \"336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821\": container with ID starting with 336ed048c5bc020b41a176ab557144cff26782f8ded0ceca171a9b78800c9821 not found: ID does not exist"
Oct 30 04:15:19.950516 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.950515    4334 scope.go:122] "RemoveContainer" containerID="87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd"
Oct 30 04:15:19.950776 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.950757    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd"} err="failed to get container status \"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd\": rpc error: code = NotFound desc = could not find container \"87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd\": container with ID starting with 87a2674f5c1cdc0eef0057e56af3a2a4e3d4ae00f8258c8afd4875820843b1dd not found: ID does not exist"
Oct 30 04:15:19.950823 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.950777    4334 scope.go:122] "RemoveContainer" containerID="91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3"
Oct 30 04:15:19.951002 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.950982    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3"} err="failed to get container status \"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3\": rpc error: code = NotFound desc = could not find container \"91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3\": container with ID starting with 91343d078fc2fd8fd94b9426145fde25b54ae8e08572d4e966ea0f4fcd88cae3 not found: ID does not exist"
Oct 30 04:15:19.951069 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.951003    4334 scope.go:122] "RemoveContainer" containerID="0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"
Oct 30 04:15:19.958390 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.958362    4334 scope.go:122] "RemoveContainer" containerID="a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"
Oct 30 04:15:19.966002 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.965983    4334 scope.go:122] "RemoveContainer" containerID="31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"
Oct 30 04:15:19.976127 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.976108    4334 scope.go:122] "RemoveContainer" containerID="b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"
Oct 30 04:15:19.984022 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.984006    4334 scope.go:122] "RemoveContainer" containerID="0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"
Oct 30 04:15:19.984276 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:19.984256    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5\": container with ID starting with 0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5 not found: ID does not exist" containerID="0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"
Oct 30 04:15:19.984389 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.984282    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"} err="failed to get container status \"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5\": rpc error: code = NotFound desc = could not find container \"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5\": container with ID starting with 0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5 not found: ID does not exist"
Oct 30 04:15:19.984389 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.984305    4334 scope.go:122] "RemoveContainer" containerID="a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"
Oct 30 04:15:19.984572 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:19.984556    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11\": container with ID starting with a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11 not found: ID does not exist" containerID="a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"
Oct 30 04:15:19.984703 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.984575    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"} err="failed to get container status \"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11\": rpc error: code = NotFound desc = could not find container \"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11\": container with ID starting with a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11 not found: ID does not exist"
Oct 30 04:15:19.984703 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.984590    4334 scope.go:122] "RemoveContainer" containerID="31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"
Oct 30 04:15:19.984823 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:19.984811    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec\": container with ID starting with 31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec not found: ID does not exist" containerID="31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"
Oct 30 04:15:19.984884 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.984827    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"} err="failed to get container status \"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec\": rpc error: code = NotFound desc = could not find container \"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec\": container with ID starting with 31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec not found: ID does not exist"
Oct 30 04:15:19.984884 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.984841    4334 scope.go:122] "RemoveContainer" containerID="b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"
Oct 30 04:15:19.985083 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:19.985065    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e\": container with ID starting with b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e not found: ID does not exist" containerID="b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"
Oct 30 04:15:19.985164 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.985087    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"} err="failed to get container status \"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e\": rpc error: code = NotFound desc = could not find container \"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e\": container with ID starting with b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e not found: ID does not exist"
Oct 30 04:15:19.985164 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.985102    4334 scope.go:122] "RemoveContainer" containerID="0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"
Oct 30 04:15:19.985324 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.985303    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"} err="failed to get container status \"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5\": rpc error: code = NotFound desc = could not find container \"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5\": container with ID starting with 0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5 not found: ID does not exist"
Oct 30 04:15:19.985324 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.985323    4334 scope.go:122] "RemoveContainer" containerID="a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"
Oct 30 04:15:19.985623 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.985602    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"} err="failed to get container status \"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11\": rpc error: code = NotFound desc = could not find container \"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11\": container with ID starting with a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11 not found: ID does not exist"
Oct 30 04:15:19.985623 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.985622    4334 scope.go:122] "RemoveContainer" containerID="31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"
Oct 30 04:15:19.985857 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.985834    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"} err="failed to get container status \"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec\": rpc error: code = NotFound desc = could not find container \"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec\": container with ID starting with 31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec not found: ID does not exist"
Oct 30 04:15:19.985935 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.985857    4334 scope.go:122] "RemoveContainer" containerID="b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"
Oct 30 04:15:19.986068 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.986051    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"} err="failed to get container status \"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e\": rpc error: code = NotFound desc = could not find container \"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e\": container with ID starting with b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e not found: ID does not exist"
Oct 30 04:15:19.986068 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.986067    4334 scope.go:122] "RemoveContainer" containerID="0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"
Oct 30 04:15:19.986272 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.986255    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5"} err="failed to get container status \"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5\": rpc error: code = NotFound desc = could not find container \"0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5\": container with ID starting with 0fad4933dca388b8559ede6899835eb4d74a8ee20e1224ac499ecdd6543575b5 not found: ID does not exist"
Oct 30 04:15:19.986340 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.986271    4334 scope.go:122] "RemoveContainer" containerID="a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"
Oct 30 04:15:19.986539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.986521    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11"} err="failed to get container status \"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11\": rpc error: code = NotFound desc = could not find container \"a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11\": container with ID starting with a36252d6066a297b79d2c7a0a73b31003bb57e6404cce7e6cb15e223628b7f11 not found: ID does not exist"
Oct 30 04:15:19.986539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.986537    4334 scope.go:122] "RemoveContainer" containerID="31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"
Oct 30 04:15:19.986801 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.986782    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec"} err="failed to get container status \"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec\": rpc error: code = NotFound desc = could not find container \"31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec\": container with ID starting with 31b54e29b30e3fb8162a37bf382ed32d515c0a6bb4fe5f15327914f16e4314ec not found: ID does not exist"
Oct 30 04:15:19.986801 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.986799    4334 scope.go:122] "RemoveContainer" containerID="b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"
Oct 30 04:15:19.987044 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.987025    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e"} err="failed to get container status \"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e\": rpc error: code = NotFound desc = could not find container \"b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e\": container with ID starting with b43c8184b6d971faa1d15947291bd055409472c884255ab2455198d8659f704e not found: ID does not exist"
Oct 30 04:15:19.999766 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.999744    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-kube-api-access-9wp4j\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-kube-api-access-9wp4j\") pod \"e2a25202-a0a3-4517-a73a-311babf85515\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") "
Oct 30 04:15:19.999845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.999799    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-config-volume\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-config-volume\") pod \"e2a25202-a0a3-4517-a73a-311babf85515\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") "
Oct 30 04:15:19.999845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.999827    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-cluster-tls-config\") pod \"e2a25202-a0a3-4517-a73a-311babf85515\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") "
Oct 30 04:15:19.999938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.999844    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-config-out\") pod \"e2a25202-a0a3-4517-a73a-311babf85515\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") "
Oct 30 04:15:19.999938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.999882    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-web-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-web-config\") pod \"e2a25202-a0a3-4517-a73a-311babf85515\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") "
Oct 30 04:15:20.000052 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.999934    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-tls-assets\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-tls-assets\") pod \"e2a25202-a0a3-4517-a73a-311babf85515\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") "
Oct 30 04:15:20.000052 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:19.999974    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-alertmanager-data-science-monitoringstack-db\") pod \"e2a25202-a0a3-4517-a73a-311babf85515\" (UID: \"e2a25202-a0a3-4517-a73a-311babf85515\") "
Oct 30 04:15:20.000316 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000295    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.000399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000334    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:15:20.000474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000400    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4526ca2d-c26f-4d26-b646-83edefe10f53-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.000474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000417    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.000474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000432    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24bmw\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-kube-api-access-24bmw\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000494    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000507    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000520    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000534    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000548    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000561    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4526ca2d-c26f-4d26-b646-83edefe10f53-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000575    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4526ca2d-c26f-4d26-b646-83edefe10f53-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000589    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4526ca2d-c26f-4d26-b646-83edefe10f53-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000677    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"]
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.000847    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.000878    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-alertmanager-data-science-monitoringstack-db" pod "e2a25202-a0a3-4517-a73a-311babf85515" (UID: "e2a25202-a0a3-4517-a73a-311babf85515"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.000927    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal podName:d664440a-1f61-4b27-9f21-53d08f8b37c0 nodeName:}" failed. No retries permitted until 2026-10-30 04:15:20.500906051 +0000 UTC m=+5599.507496295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal") pod "data-science-collector-collector-0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0") : configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.001197    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:15:20.001972 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.001269    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs podName:d664440a-1f61-4b27-9f21-53d08f8b37c0 nodeName:}" failed. No retries permitted until 2026-10-30 04:15:20.501250832 +0000 UTC m=+5599.507841098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0") : secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:15:20.003127 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.003103    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-kube-api-access-9wp4j" pod "e2a25202-a0a3-4517-a73a-311babf85515" (UID: "e2a25202-a0a3-4517-a73a-311babf85515"). InnerVolumeSpecName "kube-api-access-9wp4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:20.003243 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.003131    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-config-volume" pod "e2a25202-a0a3-4517-a73a-311babf85515" (UID: "e2a25202-a0a3-4517-a73a-311babf85515"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:20.003543 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.003519    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-config-out" pod "e2a25202-a0a3-4517-a73a-311babf85515" (UID: "e2a25202-a0a3-4517-a73a-311babf85515"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:20.005374 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.005222    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-tls-assets" pod "e2a25202-a0a3-4517-a73a-311babf85515" (UID: "e2a25202-a0a3-4517-a73a-311babf85515"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:20.007386 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.007358    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zbz7j"]
Oct 30 04:15:20.011254 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.011183    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-cluster-tls-config" pod "e2a25202-a0a3-4517-a73a-311babf85515" (UID: "e2a25202-a0a3-4517-a73a-311babf85515"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:20.015237 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.015218    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-9689c530-7f91-4584-b28e-8809bed9e799" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:15:20.018825 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.018805    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-web-config" pod "e2a25202-a0a3-4517-a73a-311babf85515" (UID: "e2a25202-a0a3-4517-a73a-311babf85515"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:20.101762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.101738    4334 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-cluster-tls-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.101762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.101760    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.101922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.101769    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.101922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.101778    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.101922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.101787    4334 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/e2a25202-a0a3-4517-a73a-311babf85515-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.101922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.101796    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wp4j\" (UniqueName: \"kubernetes.io/projected/e2a25202-a0a3-4517-a73a-311babf85515-kube-api-access-9wp4j\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.101922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.101805    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-9689c530-7f91-4584-b28e-8809bed9e799\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff90eace93e72a87\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-e364dd4f86bda642cef8d14035ca0bfc3cdafe8ca20f42d905720eb71a97db6d\""
Oct 30 04:15:20.101922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.101815    4334 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2a25202-a0a3-4517-a73a-311babf85515-config-volume\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.262861 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.262793    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:15:20.266501 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.266477    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:15:20.505583 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.505556    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:15:20.505583 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.505578    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 04:15:20.505786 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.505626    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs podName:d664440a-1f61-4b27-9f21-53d08f8b37c0 nodeName:}" failed. No retries permitted until 2026-10-30 04:15:21.505608051 +0000 UTC m=+5600.512198296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0") : secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:15:20.505786 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.505646    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal podName:d664440a-1f61-4b27-9f21-53d08f8b37c0 nodeName:}" failed. No retries permitted until 2026-10-30 04:15:21.505637716 +0000 UTC m=+5600.512227949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal") pod "data-science-collector-collector-0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0") : configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 04:15:20.925949 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.925915    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:20.927022 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.926990    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d664440a-1f61-4b27-9f21-53d08f8b37c0" containerName="otc-container" containerID="cri-o://fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662" gracePeriod=30
Oct 30 04:15:20.974458 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.974422    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:15:20.978287 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:20.978259    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:15:20.979172 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:20.979154    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:15:21.160113 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.160085    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b37f7f4-0ed2-4839-99cc-e91b670eafc7" path="/var/lib/kubelet/pods/0b37f7f4-0ed2-4839-99cc-e91b670eafc7/volumes"
Oct 30 04:15:21.160641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.160615    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" path="/var/lib/kubelet/pods/4526ca2d-c26f-4d26-b646-83edefe10f53/volumes"
Oct 30 04:15:21.161151 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.161135    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a25202-a0a3-4517-a73a-311babf85515" path="/var/lib/kubelet/pods/e2a25202-a0a3-4517-a73a-311babf85515/volumes"
Oct 30 04:15:21.167899 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.167877    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:21.312761 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.312693    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs\") pod \"d664440a-1f61-4b27-9f21-53d08f8b37c0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") "
Oct 30 04:15:21.312761 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.312727    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d664440a-1f61-4b27-9f21-53d08f8b37c0-kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/d664440a-1f61-4b27-9f21-53d08f8b37c0-kube-api-access-d97wh\") pod \"d664440a-1f61-4b27-9f21-53d08f8b37c0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") "
Oct 30 04:15:21.312937 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.312784    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-monitoring-tls-certs\") pod \"d664440a-1f61-4b27-9f21-53d08f8b37c0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") "
Oct 30 04:15:21.312937 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.312822    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal\") pod \"d664440a-1f61-4b27-9f21-53d08f8b37c0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") "
Oct 30 04:15:21.312937 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.312891    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-tls-certs\") pod \"d664440a-1f61-4b27-9f21-53d08f8b37c0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") "
Oct 30 04:15:21.313061 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.312950    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-data-science-collector-ta-client-cert\") pod \"d664440a-1f61-4b27-9f21-53d08f8b37c0\" (UID: \"d664440a-1f61-4b27-9f21-53d08f8b37c0\") "
Oct 30 04:15:21.313279 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.313242    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal" pod "d664440a-1f61-4b27-9f21-53d08f8b37c0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:21.315038 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.315011    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d664440a-1f61-4b27-9f21-53d08f8b37c0-kube-api-access-d97wh" pod "d664440a-1f61-4b27-9f21-53d08f8b37c0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0"). InnerVolumeSpecName "kube-api-access-d97wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:21.315158 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.315016    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs" pod "d664440a-1f61-4b27-9f21-53d08f8b37c0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:21.315158 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.315083    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-tls-certs" pod "d664440a-1f61-4b27-9f21-53d08f8b37c0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:21.315368 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.315349    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-monitoring-tls-certs" pod "d664440a-1f61-4b27-9f21-53d08f8b37c0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:21.315419 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.315396    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-data-science-collector-ta-client-cert" pod "d664440a-1f61-4b27-9f21-53d08f8b37c0" (UID: "d664440a-1f61-4b27-9f21-53d08f8b37c0"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:21.414077 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.414047    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:21.414077 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.414074    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:21.414223 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.414085    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97wh\" (UniqueName: \"kubernetes.io/projected/d664440a-1f61-4b27-9f21-53d08f8b37c0-kube-api-access-d97wh\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:21.414223 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.414093    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:21.414223 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.414102    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d664440a-1f61-4b27-9f21-53d08f8b37c0-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:21.414223 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.414110    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d664440a-1f61-4b27-9f21-53d08f8b37c0-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:21.935320 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.935288    4334 generic.go:356] "Generic (PLEG): container finished" podID="d664440a-1f61-4b27-9f21-53d08f8b37c0" containerID="fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662" exitCode=0
Oct 30 04:15:21.935892 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.935334    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:21.935892 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.935357    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d664440a-1f61-4b27-9f21-53d08f8b37c0","Type":"ContainerDied","Data":"fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662"}
Oct 30 04:15:21.935892 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.935397    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d664440a-1f61-4b27-9f21-53d08f8b37c0","Type":"ContainerDied","Data":"71b8f7eb0881cf41ff4e356218a7bf1a8011df38deb50499d367bea35a3a9c89"}
Oct 30 04:15:21.935892 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.935417    4334 scope.go:122] "RemoveContainer" containerID="fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662"
Oct 30 04:15:21.947716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.945714    4334 scope.go:122] "RemoveContainer" containerID="fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662"
Oct 30 04:15:21.947716 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:21.946025    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662\": container with ID starting with fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662 not found: ID does not exist" containerID="fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662"
Oct 30 04:15:21.947716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.946052    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662"} err="failed to get container status \"fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662\": rpc error: code = NotFound desc = could not find container \"fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662\": container with ID starting with fd69337ed703fd020cc629bf3fb438ebc00f9103c7b79d076fa35910c09cf662 not found: ID does not exist"
Oct 30 04:15:21.981934 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.981909    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:21.984977 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:21.984954    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:23.158206 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:23.158171    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d664440a-1f61-4b27-9f21-53d08f8b37c0" path="/var/lib/kubelet/pods/d664440a-1f61-4b27-9f21-53d08f8b37c0/volumes"
Oct 30 04:15:25.616310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616282    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"]
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616611    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="init-config-reloader"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616625    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="init-config-reloader"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616642    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="config-reloader"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616648    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="config-reloader"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616659    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="prometheus"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616664    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="prometheus"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616675    4334 cpu_manager.go:414] "containerMap: removing container" podUID="d664440a-1f61-4b27-9f21-53d08f8b37c0" containerName="otc-container"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616680    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d664440a-1f61-4b27-9f21-53d08f8b37c0" containerName="otc-container"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616694    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="alertmanager"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616700    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="alertmanager"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616712    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="init-config-reloader"
Oct 30 04:15:25.616715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616719    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="init-config-reloader"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616728    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="config-reloader"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616733    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="config-reloader"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616742    4334 cpu_manager.go:414] "containerMap: removing container" podUID="0b37f7f4-0ed2-4839-99cc-e91b670eafc7" containerName="thanos-querier"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616746    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b37f7f4-0ed2-4839-99cc-e91b670eafc7" containerName="thanos-querier"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616752    4334 cpu_manager.go:414] "containerMap: removing container" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="thanos-sidecar"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616756    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="thanos-sidecar"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616820    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b37f7f4-0ed2-4839-99cc-e91b670eafc7" containerName="thanos-querier"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616831    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="thanos-sidecar"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616845    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="config-reloader"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616855    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="prometheus"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616864    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="d664440a-1f61-4b27-9f21-53d08f8b37c0" containerName="otc-container"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616875    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a25202-a0a3-4517-a73a-311babf85515" containerName="alertmanager"
Oct 30 04:15:25.617188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.616884    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="4526ca2d-c26f-4d26-b646-83edefe10f53" containerName="config-reloader"
Oct 30 04:15:25.621097 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.621083    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.635879 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.635814    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:15:25.647011 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.646987    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:15:25.647163 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.647097    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:15:25.647163 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.647098    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:15:25.647163 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.647141    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\""
Oct 30 04:15:25.647333 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.647305    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\""
Oct 30 04:15:25.647397 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.647363    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-7hdgd\""
Oct 30 04:15:25.648969 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.648948    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"]
Oct 30 04:15:25.659738 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.659718    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"]
Oct 30 04:15:25.662937 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.662920    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.679599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.679579    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"]
Oct 30 04:15:25.698002 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.697974    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"]
Oct 30 04:15:25.701356 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.701336    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.719385 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.719362    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"]
Oct 30 04:15:25.736639 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.736610    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 04:15:25.736639 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.736625    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 04:15:25.736802 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.736644    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-s69r9\""
Oct 30 04:15:25.736802 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.736613    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 04:15:25.736802 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.736633    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-qjgdw\""
Oct 30 04:15:25.736802 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.736684    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 04:15:25.736802 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.736735    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 04:15:25.747236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747213    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7b68\" (UniqueName: \"kubernetes.io/projected/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-kube-api-access-q7b68\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.747339 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747241    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.747339 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747267    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdgxq\" (UniqueName: \"kubernetes.io/projected/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-api-access-qdgxq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.747475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747398    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.747475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747441    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.747559 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747529    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.747592 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747562    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.747638 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747593    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.747710 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747680    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.747760 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.747708    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.848514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848473    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdgxq\" (UniqueName: \"kubernetes.io/projected/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-api-access-qdgxq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.848702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848538    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3482d570-36ef-4f74-bb80-e6bff98eb3c1-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.848702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848577    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.848702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848607    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.848702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848674    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3482d570-36ef-4f74-bb80-e6bff98eb3c1-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.848958 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848724    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.848958 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848783    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.848958 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848822    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.848958 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848855    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt49t\" (UniqueName: \"kubernetes.io/projected/3482d570-36ef-4f74-bb80-e6bff98eb3c1-kube-api-access-nt49t\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.848958 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848908    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.848958 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848933    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.849256 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.848989    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7b68\" (UniqueName: \"kubernetes.io/projected/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-kube-api-access-q7b68\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.849256 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.849019    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.849371 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.849304    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.849371 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:25.849351    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found
Oct 30 04:15:25.849494 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:25.849419    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert podName:e06a50d5-234c-4074-bd93-bf9462a8083a nodeName:}" failed. No retries permitted until 2026-10-30 04:15:26.349399064 +0000 UTC m=+5605.355989309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-8sppf" (UID: "e06a50d5-234c-4074-bd93-bf9462a8083a") : secret "prometheus-operated-tls" not found
Oct 30 04:15:25.849570 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.849508    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.849764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.849743    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.850094 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.850072    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.851397 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.851370    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.851507 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.851410    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.851507 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.851431    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.867728 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.867680    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdgxq\" (UniqueName: \"kubernetes.io/projected/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-api-access-qdgxq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:25.868004 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.867981    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7b68\" (UniqueName: \"kubernetes.io/projected/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-kube-api-access-q7b68\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2z9z5\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:25.949789 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.949755    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3482d570-36ef-4f74-bb80-e6bff98eb3c1-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.949946 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.949811    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3482d570-36ef-4f74-bb80-e6bff98eb3c1-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.949946 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.949857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt49t\" (UniqueName: \"kubernetes.io/projected/3482d570-36ef-4f74-bb80-e6bff98eb3c1-kube-api-access-nt49t\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.950505 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.950481    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3482d570-36ef-4f74-bb80-e6bff98eb3c1-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.952147 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.952126    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3482d570-36ef-4f74-bb80-e6bff98eb3c1-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.968584 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.968566    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt49t\" (UniqueName: \"kubernetes.io/projected/3482d570-36ef-4f74-bb80-e6bff98eb3c1-kube-api-access-nt49t\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pd9fp\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:25.977365 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:25.977347    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:15:26.015709 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.015673    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:15:26.287900 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.287854    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"]
Oct 30 04:15:26.292427 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:26.292397    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f6dbf0_96a6_43a9_a4c1_bdfe011cec7f.slice/crio-7d9cb7818ca11e6997b999539d77c08ff3d924d5b37ab76550a383403b9c568e WatchSource:0}: Error finding container 7d9cb7818ca11e6997b999539d77c08ff3d924d5b37ab76550a383403b9c568e: Status 404 returned error can't find the container with id 7d9cb7818ca11e6997b999539d77c08ff3d924d5b37ab76550a383403b9c568e
Oct 30 04:15:26.325824 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.325799    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"]
Oct 30 04:15:26.329411 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:26.329383    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3482d570_36ef_4f74_bb80_e6bff98eb3c1.slice/crio-1cc2a3b5428221dc79b068a79220a3e25dff92e09d7e135c98d1f168be8835aa WatchSource:0}: Error finding container 1cc2a3b5428221dc79b068a79220a3e25dff92e09d7e135c98d1f168be8835aa: Status 404 returned error can't find the container with id 1cc2a3b5428221dc79b068a79220a3e25dff92e09d7e135c98d1f168be8835aa
Oct 30 04:15:26.353841 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.353748    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:26.356430 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.356406    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8sppf\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:26.533256 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.533232    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:26.829819 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.829742    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"]
Oct 30 04:15:26.833861 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:26.833832    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06a50d5_234c_4074_bd93_bf9462a8083a.slice/crio-bc8ea93b3b517089366dbf77aca2df112062b766741f44b749e148bee05f10a8 WatchSource:0}: Error finding container bc8ea93b3b517089366dbf77aca2df112062b766741f44b749e148bee05f10a8: Status 404 returned error can't find the container with id bc8ea93b3b517089366dbf77aca2df112062b766741f44b749e148bee05f10a8
Oct 30 04:15:26.960116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.960086    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf" event={"ID":"e06a50d5-234c-4074-bd93-bf9462a8083a","Type":"ContainerStarted","Data":"bc8ea93b3b517089366dbf77aca2df112062b766741f44b749e148bee05f10a8"}
Oct 30 04:15:26.961864 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.961840    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" event={"ID":"3482d570-36ef-4f74-bb80-e6bff98eb3c1","Type":"ContainerStarted","Data":"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6"}
Oct 30 04:15:26.962000 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.961893    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" event={"ID":"3482d570-36ef-4f74-bb80-e6bff98eb3c1","Type":"ContainerStarted","Data":"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113"}
Oct 30 04:15:26.962000 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.961916    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" event={"ID":"3482d570-36ef-4f74-bb80-e6bff98eb3c1","Type":"ContainerStarted","Data":"1cc2a3b5428221dc79b068a79220a3e25dff92e09d7e135c98d1f168be8835aa"}
Oct 30 04:15:26.963700 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.963677    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" event={"ID":"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f","Type":"ContainerStarted","Data":"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8"}
Oct 30 04:15:26.963816 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.963718    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" event={"ID":"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f","Type":"ContainerStarted","Data":"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4"}
Oct 30 04:15:26.963816 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.963736    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" event={"ID":"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f","Type":"ContainerStarted","Data":"7d9cb7818ca11e6997b999539d77c08ff3d924d5b37ab76550a383403b9c568e"}
Oct 30 04:15:26.999662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:26.999618    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" podStartSLOduration=1.999604557 podStartE2EDuration="1.999604557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:26.998173906 +0000 UTC m=+5606.004764160" watchObservedRunningTime="2026-10-30 04:15:26.999604557 +0000 UTC m=+5606.006194790"
Oct 30 04:15:27.037733 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.037676    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" podStartSLOduration=2.037662615 podStartE2EDuration="2.037662615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:27.036824854 +0000 UTC m=+5606.043415108" watchObservedRunningTime="2026-10-30 04:15:27.037662615 +0000 UTC m=+5606.044252865"
Oct 30 04:15:27.107229 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.107196    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:27.111810 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.111790    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.129506 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.129480    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:27.130163 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.129823    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:15:27.130163 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.129948    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:15:27.130163 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.130031    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:15:27.130163 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.130129    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:15:27.137518 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.137493    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:15:27.264663 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.264608    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52c43dcf-a2fc-4105-b50a-673cf25830b1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.264844 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.264683    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.264844 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.264783    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.264844 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.264833    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.265054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.265034    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.265139 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.265119    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq9ks\" (UniqueName: \"kubernetes.io/projected/52c43dcf-a2fc-4105-b50a-673cf25830b1-kube-api-access-vq9ks\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.365804 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.365723    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.365804 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.365784    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9ks\" (UniqueName: \"kubernetes.io/projected/52c43dcf-a2fc-4105-b50a-673cf25830b1-kube-api-access-vq9ks\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.366134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.365826    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52c43dcf-a2fc-4105-b50a-673cf25830b1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.366134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.365857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.366134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.365969    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.366134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.366033    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.366911 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.366882    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52c43dcf-a2fc-4105-b50a-673cf25830b1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.368614 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.368582    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.368758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.368587    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.368758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.368619    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.368889 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.368839    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.387781 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.387756    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9ks\" (UniqueName: \"kubernetes.io/projected/52c43dcf-a2fc-4105-b50a-673cf25830b1-kube-api-access-vq9ks\") pod \"data-science-collector-collector-0\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.426521 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.426488    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:27.725538 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.725505    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:27.729542 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:27.729516    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c43dcf_a2fc_4105_b50a_673cf25830b1.slice/crio-6fffdf3dc4fdb8873a825110859cd1dda814d7b58ce4fcaf8943ca1d716e0f8d WatchSource:0}: Error finding container 6fffdf3dc4fdb8873a825110859cd1dda814d7b58ce4fcaf8943ca1d716e0f8d: Status 404 returned error can't find the container with id 6fffdf3dc4fdb8873a825110859cd1dda814d7b58ce4fcaf8943ca1d716e0f8d
Oct 30 04:15:27.731652 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.731620    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:15:27.731751 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.731676    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:15:27.970522 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.970487    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"52c43dcf-a2fc-4105-b50a-673cf25830b1","Type":"ContainerStarted","Data":"fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466"}
Oct 30 04:15:27.970933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.970547    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"52c43dcf-a2fc-4105-b50a-673cf25830b1","Type":"ContainerStarted","Data":"6fffdf3dc4fdb8873a825110859cd1dda814d7b58ce4fcaf8943ca1d716e0f8d"}
Oct 30 04:15:27.972102 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:27.972075    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf" event={"ID":"e06a50d5-234c-4074-bd93-bf9462a8083a","Type":"ContainerStarted","Data":"3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a"}
Oct 30 04:15:28.012082 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:28.012006    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.011994607 podStartE2EDuration="1.011994607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:28.011284977 +0000 UTC m=+5607.017875232" watchObservedRunningTime="2026-10-30 04:15:28.011994607 +0000 UTC m=+5607.018584940"
Oct 30 04:15:28.049284 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:28.049231    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf" podStartSLOduration=3.049216553 podStartE2EDuration="3.049216553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:28.048041264 +0000 UTC m=+5607.054631518" watchObservedRunningTime="2026-10-30 04:15:28.049216553 +0000 UTC m=+5607.055806862"
Oct 30 04:15:28.977847 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:28.977814    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:30.768357 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.768309    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:15:30.772241 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.772215    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.803686 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.803659    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:15:30.822107 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.822074    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:15:30.826183 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.826166    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.848968 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.848940    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:15:30.861473 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861423    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:15:30.861649 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861501    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:15:30.861649 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861518    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:15:30.861762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861737    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:15:30.861762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861757    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:15:30.861762 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861758    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:15:30.861963 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861774    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:15:30.861963 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861789    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:15:30.861963 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861849    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-8vqvg\""
Oct 30 04:15:30.861963 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861879    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:15:30.861963 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.861944    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-dwdtj\""
Oct 30 04:15:30.896366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.896334    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.896366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.896378    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcjf\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-kube-api-access-ppcjf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.896643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.896412    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.896643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.896491    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.896643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.896597    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.896643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.896641    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.896848 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.896731    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.905389 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.905361    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:15:30.905544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.905489    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:15:30.905544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.905503    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:15:30.997953 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.997918    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998144 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.997958    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998144 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.997992    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.998144 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998035    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.998144 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998117    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998352 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998235    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998352 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998270    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.998352 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998322    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998594 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998351    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998594 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998372    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dnb\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-kube-api-access-z8dnb\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998594 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998422    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.998594 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998485    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcjf\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-kube-api-access-ppcjf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.998594 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998514    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.998594 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998538    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998594 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998563    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.998908 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998797    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998908 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998828    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.998908 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:30.999037 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998911    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:30.999037 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:30.998953    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.000666 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.000641    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:31.001016 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.000992    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:31.001127 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.001105    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:31.001212 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.001196    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:31.001289 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.001271    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:31.015888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.015866    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcjf\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-kube-api-access-ppcjf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:31.092592 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.092514    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:31.099722 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.099701    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.099798 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.099753    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.099798 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.099776    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.099880 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.099819    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.099880 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.099857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dnb\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-kube-api-access-z8dnb\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.099977 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.099914    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.099977 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.099955    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.100089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.099991    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.100089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.100025    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.100089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.100060    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.100252 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.100094    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.100252 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.100122    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.100430 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.100403    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.101022 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.100997    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.102921 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.102753    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.103036 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.102931    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.103268 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.103224    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.103820 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.103783    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.103820 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.103793    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.103934 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.103857    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.104365 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.104340    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.104513 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.104496    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.104745 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.104728    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.118132 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.118112    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dnb\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-kube-api-access-z8dnb\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:31.403779 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.403702    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:15:31.409256 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:31.409231    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3a34b7_a892_43fc_a24f_15e575f90960.slice/crio-ddc523d20860e3dee663efaf8e1adec3d5871f5a31dcbabe056e6637643aa36d WatchSource:0}: Error finding container ddc523d20860e3dee663efaf8e1adec3d5871f5a31dcbabe056e6637643aa36d: Status 404 returned error can't find the container with id ddc523d20860e3dee663efaf8e1adec3d5871f5a31dcbabe056e6637643aa36d
Oct 30 04:15:31.998409 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.998327    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ff3a34b7-a892-43fc-a24f-15e575f90960","Type":"ContainerStarted","Data":"78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8"}
Oct 30 04:15:31.998409 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:31.998382    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ff3a34b7-a892-43fc-a24f-15e575f90960","Type":"ContainerStarted","Data":"ddc523d20860e3dee663efaf8e1adec3d5871f5a31dcbabe056e6637643aa36d"}
Oct 30 04:15:34.986866 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:34.986838    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:15:36.018369 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:36.018333    4334 generic.go:356] "Generic (PLEG): container finished" podID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerID="78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8" exitCode=0
Oct 30 04:15:36.018771 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:36.018394    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ff3a34b7-a892-43fc-a24f-15e575f90960","Type":"ContainerDied","Data":"78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8"}
Oct 30 04:15:37.031535 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:37.031496    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ff3a34b7-a892-43fc-a24f-15e575f90960","Type":"ContainerStarted","Data":"5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4"}
Oct 30 04:15:37.031950 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:37.031590    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ff3a34b7-a892-43fc-a24f-15e575f90960","Type":"ContainerStarted","Data":"a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525"}
Oct 30 04:15:37.031950 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:37.031882    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:37.094507 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:37.094383    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.709493798 podStartE2EDuration="7.094372213s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384878415s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:37.082597758 +0000 UTC m=+5616.089188014" watchObservedRunningTime="2026-10-30 04:15:37.094372213 +0000 UTC m=+5616.100962467"
Oct 30 04:15:38.358648 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.358613    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:38.385381 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.385353    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:38.459881 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.459857    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:38.460005 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.459968    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:38.472979 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.472956    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:38.575126 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.575089    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:38.600287 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.600264    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:38.646562 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.646508    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:38.973690 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:38.973487    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:15:38.976399 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:38.976368    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ec58dd_c905_47f1_a2f7_66b7a77ffa41.slice/crio-baa0de2989eb0456a1b592bdbb7497673959fd9c4f130e163554a9b4b73d406c WatchSource:0}: Error finding container baa0de2989eb0456a1b592bdbb7497673959fd9c4f130e163554a9b4b73d406c: Status 404 returned error can't find the container with id baa0de2989eb0456a1b592bdbb7497673959fd9c4f130e163554a9b4b73d406c
Oct 30 04:15:39.042561 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:39.042534    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerStarted","Data":"baa0de2989eb0456a1b592bdbb7497673959fd9c4f130e163554a9b4b73d406c"}
Oct 30 04:15:40.048833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:40.048793    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerStarted","Data":"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"}
Oct 30 04:15:43.037706 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:43.037616    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:15:43.065575 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:43.065548    4334 generic.go:356] "Generic (PLEG): container finished" podID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerID="2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8" exitCode=0
Oct 30 04:15:43.065718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:43.065604    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerDied","Data":"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"}
Oct 30 04:15:44.073393 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:44.073358    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerStarted","Data":"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"}
Oct 30 04:15:44.073776 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:44.073422    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerStarted","Data":"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"}
Oct 30 04:15:44.073776 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:44.073466    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerStarted","Data":"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"}
Oct 30 04:15:44.129907 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:44.129860    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.74493554 podStartE2EDuration="14.129847166s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384911626s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:44.11781878 +0000 UTC m=+5623.124409033" watchObservedRunningTime="2026-10-30 04:15:44.129847166 +0000 UTC m=+5623.136437419"
Oct 30 04:15:48.647585 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:48.647550    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:49.399553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:49.384491    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:49.399553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:49.384735    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="52c43dcf-a2fc-4105-b50a-673cf25830b1" containerName="otc-container" containerID="cri-o://fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466" gracePeriod=30
Oct 30 04:15:50.633974 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.633950    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:50.663336 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.663313    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-data-science-collector-ta-client-cert\") pod \"52c43dcf-a2fc-4105-b50a-673cf25830b1\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") "
Oct 30 04:15:50.663470 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.663408    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-otlp-tls-certs\") pod \"52c43dcf-a2fc-4105-b50a-673cf25830b1\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") "
Oct 30 04:15:50.663470 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.663430    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-monitoring-tls-certs\") pod \"52c43dcf-a2fc-4105-b50a-673cf25830b1\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") "
Oct 30 04:15:50.663470 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.663459    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c43dcf-a2fc-4105-b50a-673cf25830b1-kube-api-access-vq9ks\" (UniqueName: \"kubernetes.io/projected/52c43dcf-a2fc-4105-b50a-673cf25830b1-kube-api-access-vq9ks\") pod \"52c43dcf-a2fc-4105-b50a-673cf25830b1\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") "
Oct 30 04:15:50.663626 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.663491    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c43dcf-a2fc-4105-b50a-673cf25830b1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/52c43dcf-a2fc-4105-b50a-673cf25830b1-otc-internal\") pod \"52c43dcf-a2fc-4105-b50a-673cf25830b1\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") "
Oct 30 04:15:50.663626 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.663524    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-tls-certs\") pod \"52c43dcf-a2fc-4105-b50a-673cf25830b1\" (UID: \"52c43dcf-a2fc-4105-b50a-673cf25830b1\") "
Oct 30 04:15:50.663902 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.663875    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c43dcf-a2fc-4105-b50a-673cf25830b1-otc-internal" pod "52c43dcf-a2fc-4105-b50a-673cf25830b1" (UID: "52c43dcf-a2fc-4105-b50a-673cf25830b1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:50.665681 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.665658    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-data-science-collector-ta-client-cert" pod "52c43dcf-a2fc-4105-b50a-673cf25830b1" (UID: "52c43dcf-a2fc-4105-b50a-673cf25830b1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:50.665804 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.665765    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c43dcf-a2fc-4105-b50a-673cf25830b1-kube-api-access-vq9ks" pod "52c43dcf-a2fc-4105-b50a-673cf25830b1" (UID: "52c43dcf-a2fc-4105-b50a-673cf25830b1"). InnerVolumeSpecName "kube-api-access-vq9ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:50.665854 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.665811    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-tls-certs" pod "52c43dcf-a2fc-4105-b50a-673cf25830b1" (UID: "52c43dcf-a2fc-4105-b50a-673cf25830b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:50.665995 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.665980    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-otlp-tls-certs" pod "52c43dcf-a2fc-4105-b50a-673cf25830b1" (UID: "52c43dcf-a2fc-4105-b50a-673cf25830b1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:50.666139 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.666125    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-monitoring-tls-certs" pod "52c43dcf-a2fc-4105-b50a-673cf25830b1" (UID: "52c43dcf-a2fc-4105-b50a-673cf25830b1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:50.765219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.765158    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:50.765219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.765182    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:50.765219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.765192    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:50.765219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.765201    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52c43dcf-a2fc-4105-b50a-673cf25830b1-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:50.765219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.765210    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq9ks\" (UniqueName: \"kubernetes.io/projected/52c43dcf-a2fc-4105-b50a-673cf25830b1-kube-api-access-vq9ks\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:50.765219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:50.765218    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52c43dcf-a2fc-4105-b50a-673cf25830b1-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:15:50.977838 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:50.977793    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:15:51.111553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.111475    4334 generic.go:356] "Generic (PLEG): container finished" podID="52c43dcf-a2fc-4105-b50a-673cf25830b1" containerID="fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466" exitCode=0
Oct 30 04:15:51.111553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.111503    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.111553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.111535    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"52c43dcf-a2fc-4105-b50a-673cf25830b1","Type":"ContainerDied","Data":"fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466"}
Oct 30 04:15:51.111764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.111591    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"52c43dcf-a2fc-4105-b50a-673cf25830b1","Type":"ContainerDied","Data":"6fffdf3dc4fdb8873a825110859cd1dda814d7b58ce4fcaf8943ca1d716e0f8d"}
Oct 30 04:15:51.111764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.111617    4334 scope.go:122] "RemoveContainer" containerID="fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466"
Oct 30 04:15:51.121205 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.121185    4334 scope.go:122] "RemoveContainer" containerID="fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466"
Oct 30 04:15:51.121511 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:15:51.121461    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466\": container with ID starting with fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466 not found: ID does not exist" containerID="fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466"
Oct 30 04:15:51.121511 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.121489    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466"} err="failed to get container status \"fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466\": rpc error: code = NotFound desc = could not find container \"fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466\": container with ID starting with fa6eea26bd8ee8d34043833ac28d400b67a1c59cbc5cd9b2e688ccda47ade466 not found: ID does not exist"
Oct 30 04:15:51.159137 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.159112    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:51.169065 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.169043    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:51.199495 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.199469    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:51.199908 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.199891    4334 cpu_manager.go:414] "containerMap: removing container" podUID="52c43dcf-a2fc-4105-b50a-673cf25830b1" containerName="otc-container"
Oct 30 04:15:51.199908 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.199908    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c43dcf-a2fc-4105-b50a-673cf25830b1" containerName="otc-container"
Oct 30 04:15:51.200066 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.199988    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c43dcf-a2fc-4105-b50a-673cf25830b1" containerName="otc-container"
Oct 30 04:15:51.207146 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.207124    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.221246 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.221206    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:15:51.221342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.221207    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9vgz\""
Oct 30 04:15:51.221342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.221262    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:15:51.221342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.221290    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:51.221562 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.221544    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:15:51.221693 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.221615    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:15:51.221693 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.221648    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:15:51.269623 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.269599    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/57a96b75-1434-4c28-a2e1-27be1bef9dae-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.269744 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.269632    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.269744 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.269673    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvssl\" (UniqueName: \"kubernetes.io/projected/57a96b75-1434-4c28-a2e1-27be1bef9dae-kube-api-access-wvssl\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.269825 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.269767    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.269825 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.269791    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.269825 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.269816    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.371140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.371081    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/57a96b75-1434-4c28-a2e1-27be1bef9dae-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.371140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.371110    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.371347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.371145    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvssl\" (UniqueName: \"kubernetes.io/projected/57a96b75-1434-4c28-a2e1-27be1bef9dae-kube-api-access-wvssl\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.371347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.371168    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.371347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.371188    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.371347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.371207    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.371847 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.371824    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/57a96b75-1434-4c28-a2e1-27be1bef9dae-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.373763 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.373740    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.373861 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.373778    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.373904 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.373859    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.374911 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.374894    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.388685 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.388663    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvssl\" (UniqueName: \"kubernetes.io/projected/57a96b75-1434-4c28-a2e1-27be1bef9dae-kube-api-access-wvssl\") pod \"data-science-collector-collector-0\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.519223 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.519203    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:15:51.812731 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.812702    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:15:51.816072 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:15:51.816039    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a96b75_1434_4c28_a2e1_27be1bef9dae.slice/crio-41e9104ea447e8ab5161856174a5d952a27db00d47e80191f7a0b53364fdaf77 WatchSource:0}: Error finding container 41e9104ea447e8ab5161856174a5d952a27db00d47e80191f7a0b53364fdaf77: Status 404 returned error can't find the container with id 41e9104ea447e8ab5161856174a5d952a27db00d47e80191f7a0b53364fdaf77
Oct 30 04:15:51.818408 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.818371    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:15:51.818480 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:51.818466    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:15:52.118700 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:52.118614    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"57a96b75-1434-4c28-a2e1-27be1bef9dae","Type":"ContainerStarted","Data":"d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59"}
Oct 30 04:15:52.118700 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:52.118676    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"57a96b75-1434-4c28-a2e1-27be1bef9dae","Type":"ContainerStarted","Data":"41e9104ea447e8ab5161856174a5d952a27db00d47e80191f7a0b53364fdaf77"}
Oct 30 04:15:52.155949 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:52.155908    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.155894754 podStartE2EDuration="1.155894754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:52.15456033 +0000 UTC m=+5631.161150584" watchObservedRunningTime="2026-10-30 04:15:52.155894754 +0000 UTC m=+5631.162485008"
Oct 30 04:15:53.158940 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:53.158905    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c43dcf-a2fc-4105-b50a-673cf25830b1" path="/var/lib/kubelet/pods/52c43dcf-a2fc-4105-b50a-673cf25830b1/volumes"
Oct 30 04:15:53.646757 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:53.646677    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:53.652243 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:53.652220    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:15:54.133962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:15:54.133934    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:02.449534 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.449440    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:02.450356 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.449673    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="57a96b75-1434-4c28-a2e1-27be1bef9dae" containerName="otc-container" containerID="cri-o://d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59" gracePeriod=30
Oct 30 04:16:02.699951 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.699897    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:02.772961 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.772929    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a96b75-1434-4c28-a2e1-27be1bef9dae-kube-api-access-wvssl\" (UniqueName: \"kubernetes.io/projected/57a96b75-1434-4c28-a2e1-27be1bef9dae-kube-api-access-wvssl\") pod \"57a96b75-1434-4c28-a2e1-27be1bef9dae\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") "
Oct 30 04:16:02.773116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.773000    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a96b75-1434-4c28-a2e1-27be1bef9dae-otc-internal\" (UniqueName: \"kubernetes.io/configmap/57a96b75-1434-4c28-a2e1-27be1bef9dae-otc-internal\") pod \"57a96b75-1434-4c28-a2e1-27be1bef9dae\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") "
Oct 30 04:16:02.773116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.773023    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-monitoring-tls-certs\") pod \"57a96b75-1434-4c28-a2e1-27be1bef9dae\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") "
Oct 30 04:16:02.773116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.773045    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-tls-certs\") pod \"57a96b75-1434-4c28-a2e1-27be1bef9dae\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") "
Oct 30 04:16:02.773116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.773062    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-data-science-collector-ta-client-cert\") pod \"57a96b75-1434-4c28-a2e1-27be1bef9dae\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") "
Oct 30 04:16:02.773116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.773079    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-otlp-tls-certs\") pod \"57a96b75-1434-4c28-a2e1-27be1bef9dae\" (UID: \"57a96b75-1434-4c28-a2e1-27be1bef9dae\") "
Oct 30 04:16:02.773590 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.773556    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a96b75-1434-4c28-a2e1-27be1bef9dae-otc-internal" pod "57a96b75-1434-4c28-a2e1-27be1bef9dae" (UID: "57a96b75-1434-4c28-a2e1-27be1bef9dae"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:02.775154 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.775127    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a96b75-1434-4c28-a2e1-27be1bef9dae-kube-api-access-wvssl" pod "57a96b75-1434-4c28-a2e1-27be1bef9dae" (UID: "57a96b75-1434-4c28-a2e1-27be1bef9dae"). InnerVolumeSpecName "kube-api-access-wvssl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:02.775329 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.775307    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-data-science-collector-ta-client-cert" pod "57a96b75-1434-4c28-a2e1-27be1bef9dae" (UID: "57a96b75-1434-4c28-a2e1-27be1bef9dae"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:02.775401 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.775359    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-otlp-tls-certs" pod "57a96b75-1434-4c28-a2e1-27be1bef9dae" (UID: "57a96b75-1434-4c28-a2e1-27be1bef9dae"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:02.775588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.775569    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-monitoring-tls-certs" pod "57a96b75-1434-4c28-a2e1-27be1bef9dae" (UID: "57a96b75-1434-4c28-a2e1-27be1bef9dae"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:02.775651 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.775626    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-tls-certs" pod "57a96b75-1434-4c28-a2e1-27be1bef9dae" (UID: "57a96b75-1434-4c28-a2e1-27be1bef9dae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:02.874730 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.874708    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvssl\" (UniqueName: \"kubernetes.io/projected/57a96b75-1434-4c28-a2e1-27be1bef9dae-kube-api-access-wvssl\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:02.874730 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.874728    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/57a96b75-1434-4c28-a2e1-27be1bef9dae-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:02.874897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.874738    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:02.874897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.874747    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:02.874897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.874755    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:02.874897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:02.874763    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a96b75-1434-4c28-a2e1-27be1bef9dae-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.175840 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.175813    4334 generic.go:356] "Generic (PLEG): container finished" podID="57a96b75-1434-4c28-a2e1-27be1bef9dae" containerID="d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59" exitCode=0
Oct 30 04:16:03.175996 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.175846    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"57a96b75-1434-4c28-a2e1-27be1bef9dae","Type":"ContainerDied","Data":"d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59"}
Oct 30 04:16:03.175996 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.175897    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.175996 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.175939    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"57a96b75-1434-4c28-a2e1-27be1bef9dae","Type":"ContainerDied","Data":"41e9104ea447e8ab5161856174a5d952a27db00d47e80191f7a0b53364fdaf77"}
Oct 30 04:16:03.175996 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.175990    4334 scope.go:122] "RemoveContainer" containerID="d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59"
Oct 30 04:16:03.184474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.184437    4334 scope.go:122] "RemoveContainer" containerID="d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59"
Oct 30 04:16:03.184729 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:03.184710    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59\": container with ID starting with d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59 not found: ID does not exist" containerID="d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59"
Oct 30 04:16:03.184783 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.184738    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59"} err="failed to get container status \"d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59\": rpc error: code = NotFound desc = could not find container \"d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59\": container with ID starting with d2d356a615ce0f8ee7ac63cedb3833b32f735f5f25f43959bfdcc05316544b59 not found: ID does not exist"
Oct 30 04:16:03.213297 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.213267    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"]
Oct 30 04:16:03.213508 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.213475    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf" podUID="e06a50d5-234c-4074-bd93-bf9462a8083a" containerName="kube-rbac-proxy" containerID="cri-o://3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a" gracePeriod=30
Oct 30 04:16:03.225750 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.225730    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:03.231888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.231868    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:03.236865 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.236845    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"]
Oct 30 04:16:03.237061 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.237029    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerName="kube-rbac-proxy" containerID="cri-o://3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4" gracePeriod=30
Oct 30 04:16:03.237140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.237125    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerName="prom-label-proxy" containerID="cri-o://1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8" gracePeriod=30
Oct 30 04:16:03.252155 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.252133    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"]
Oct 30 04:16:03.252479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.252427    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerName="kube-rbac-proxy" containerID="cri-o://aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113" gracePeriod=30
Oct 30 04:16:03.252580 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.252438    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerName="prom-label-proxy" containerID="cri-o://f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6" gracePeriod=30
Oct 30 04:16:03.265939 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.265920    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:03.266360 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.266337    4334 cpu_manager.go:414] "containerMap: removing container" podUID="57a96b75-1434-4c28-a2e1-27be1bef9dae" containerName="otc-container"
Oct 30 04:16:03.266360 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.266355    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a96b75-1434-4c28-a2e1-27be1bef9dae" containerName="otc-container"
Oct 30 04:16:03.266548 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.266432    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a96b75-1434-4c28-a2e1-27be1bef9dae" containerName="otc-container"
Oct 30 04:16:03.269516 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.269497    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.284203 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.284181    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:16:03.284339 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.284221    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:16:03.284434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.284256    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9vgz\""
Oct 30 04:16:03.284434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.284301    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:16:03.284593 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.284382    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:16:03.284593 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.284474    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:16:03.286082 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.286060    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:03.377886 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.377854    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.378042 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.377930    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.378042 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.378006    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.378042 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.378038    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwvq\" (UniqueName: \"kubernetes.io/projected/18f9e9f7-328c-41f8-8377-54c8862e6383-kube-api-access-jhwvq\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.378158 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.378068    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.378158 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.378100    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.478995 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.478965    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.479378 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.479028    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.479378 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.479050    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwvq\" (UniqueName: \"kubernetes.io/projected/18f9e9f7-328c-41f8-8377-54c8862e6383-kube-api-access-jhwvq\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.479378 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.479067    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.479378 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.479085    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.479378 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.479115    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.479906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.479882    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.482236 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.482212    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.483246 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.483192    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.484000 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.483942    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.485497 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.485470    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.500199 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.500173    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwvq\" (UniqueName: \"kubernetes.io/projected/18f9e9f7-328c-41f8-8377-54c8862e6383-kube-api-access-jhwvq\") pod \"data-science-collector-collector-0\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.525686 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.525666    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:16:03.543889 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.543805    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:16:03.548242 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.548216    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:16:03.678092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.677991    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:03.680674 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680650    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-ca\") pod \"e06a50d5-234c-4074-bd93-bf9462a8083a\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") "
Oct 30 04:16:03.680833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680686    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-config\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-config\") pod \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") "
Oct 30 04:16:03.680833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680720    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert\") pod \"e06a50d5-234c-4074-bd93-bf9462a8083a\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") "
Oct 30 04:16:03.680833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680784    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-kube-api-access-q7b68\" (UniqueName: \"kubernetes.io/projected/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-kube-api-access-q7b68\") pod \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") "
Oct 30 04:16:03.680833 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680818    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3482d570-36ef-4f74-bb80-e6bff98eb3c1-kube-api-access-nt49t\" (UniqueName: \"kubernetes.io/projected/3482d570-36ef-4f74-bb80-e6bff98eb3c1-kube-api-access-nt49t\") pod \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") "
Oct 30 04:16:03.681054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680859    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-api-access-qdgxq\" (UniqueName: \"kubernetes.io/projected/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-api-access-qdgxq\") pod \"e06a50d5-234c-4074-bd93-bf9462a8083a\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") "
Oct 30 04:16:03.681054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680889    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-client-ca\") pod \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") "
Oct 30 04:16:03.681054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680940    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3482d570-36ef-4f74-bb80-e6bff98eb3c1-config\" (UniqueName: \"kubernetes.io/configmap/3482d570-36ef-4f74-bb80-e6bff98eb3c1-config\") pod \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") "
Oct 30 04:16:03.681054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.680970    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3482d570-36ef-4f74-bb80-e6bff98eb3c1-tls\" (UniqueName: \"kubernetes.io/secret/3482d570-36ef-4f74-bb80-e6bff98eb3c1-tls\") pod \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\" (UID: \"3482d570-36ef-4f74-bb80-e6bff98eb3c1\") "
Oct 30 04:16:03.681054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.681003    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-tls\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-tls\") pod \"e06a50d5-234c-4074-bd93-bf9462a8083a\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") "
Oct 30 04:16:03.681054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.681033    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-rbac-proxy-config\") pod \"e06a50d5-234c-4074-bd93-bf9462a8083a\" (UID: \"e06a50d5-234c-4074-bd93-bf9462a8083a\") "
Oct 30 04:16:03.681521 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.681080    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-tls\" (UniqueName: \"kubernetes.io/secret/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-tls\") pod \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") "
Oct 30 04:16:03.681521 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.681110    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-prometheus-tls-ca\") pod \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\" (UID: \"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f\") "
Oct 30 04:16:03.681521 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.681171    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-config" pod "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" (UID: "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:03.682294 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.682001    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-client-ca" pod "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" (UID: "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:03.682822 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.682426    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.682822 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.682473    4334 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-client-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.682984 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.682907    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3482d570-36ef-4f74-bb80-e6bff98eb3c1-config" pod "3482d570-36ef-4f74-bb80-e6bff98eb3c1" (UID: "3482d570-36ef-4f74-bb80-e6bff98eb3c1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:03.683398 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.683349    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-ca" pod "e06a50d5-234c-4074-bd93-bf9462a8083a" (UID: "e06a50d5-234c-4074-bd93-bf9462a8083a"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:03.685098 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.685071    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-prometheus-tls-ca" pod "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" (UID: "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:03.688853 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.688833    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482d570-36ef-4f74-bb80-e6bff98eb3c1-kube-api-access-nt49t" pod "3482d570-36ef-4f74-bb80-e6bff98eb3c1" (UID: "3482d570-36ef-4f74-bb80-e6bff98eb3c1"). InnerVolumeSpecName "kube-api-access-nt49t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:03.690488 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.690418    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-tls" pod "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" (UID: "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:03.690488 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.690494    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert" pod "e06a50d5-234c-4074-bd93-bf9462a8083a" (UID: "e06a50d5-234c-4074-bd93-bf9462a8083a"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:03.690678 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.690657    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-kube-api-access-q7b68" pod "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" (UID: "b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f"). InnerVolumeSpecName "kube-api-access-q7b68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:03.690725 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.690693    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-tls" pod "e06a50d5-234c-4074-bd93-bf9462a8083a" (UID: "e06a50d5-234c-4074-bd93-bf9462a8083a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:03.690863 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.690844    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3482d570-36ef-4f74-bb80-e6bff98eb3c1-tls" pod "3482d570-36ef-4f74-bb80-e6bff98eb3c1" (UID: "3482d570-36ef-4f74-bb80-e6bff98eb3c1"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:03.690918 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.690904    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-api-access-qdgxq" pod "e06a50d5-234c-4074-bd93-bf9462a8083a" (UID: "e06a50d5-234c-4074-bd93-bf9462a8083a"). InnerVolumeSpecName "kube-api-access-qdgxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:03.691157 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.691136    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-rbac-proxy-config" pod "e06a50d5-234c-4074-bd93-bf9462a8083a" (UID: "e06a50d5-234c-4074-bd93-bf9462a8083a"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:03.705242 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.705216    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:03.705693 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.705514    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="prometheus" containerID="cri-o://80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6" gracePeriod=600
Oct 30 04:16:03.705693 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.705553    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="config-reloader" containerID="cri-o://3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5" gracePeriod=600
Oct 30 04:16:03.705693 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.705549    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="thanos-sidecar" containerID="cri-o://13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b" gracePeriod=600
Oct 30 04:16:03.717563 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.717527    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:03.717849 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.717813    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="alertmanager" containerID="cri-o://a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525" gracePeriod=120
Oct 30 04:16:03.717951 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.717898    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="config-reloader" containerID="cri-o://5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4" gracePeriod=120
Oct 30 04:16:03.783545 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783518    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3482d570-36ef-4f74-bb80-e6bff98eb3c1-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783545 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783544    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783556    4334 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-rbac-proxy-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783567    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783580    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-prometheus-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783590    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783604    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e06a50d5-234c-4074-bd93-bf9462a8083a-prometheus-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783616    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7b68\" (UniqueName: \"kubernetes.io/projected/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f-kube-api-access-q7b68\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783628    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt49t\" (UniqueName: \"kubernetes.io/projected/3482d570-36ef-4f74-bb80-e6bff98eb3c1-kube-api-access-nt49t\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783640    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdgxq\" (UniqueName: \"kubernetes.io/projected/e06a50d5-234c-4074-bd93-bf9462a8083a-kube-api-access-qdgxq\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.783727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.783653    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3482d570-36ef-4f74-bb80-e6bff98eb3c1-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:03.950004 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.949980    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:03.985039 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.985009    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:03.988243 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:16:03.988217    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f9e9f7_328c_41f8_8377_54c8862e6383.slice/crio-a1193bc3b0cd6ec8dc58eefbf2785539b08947297976df1d0675daf185d3133f WatchSource:0}: Error finding container a1193bc3b0cd6ec8dc58eefbf2785539b08947297976df1d0675daf185d3133f: Status 404 returned error can't find the container with id a1193bc3b0cd6ec8dc58eefbf2785539b08947297976df1d0675daf185d3133f
Oct 30 04:16:03.990260 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.990234    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:16:03.990320 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:03.990286    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:16:04.085857 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.085833    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.085989 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.085962    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086121 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086009    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-kube-api-access-z8dnb\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-kube-api-access-z8dnb\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086121 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086040    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config-out\" (UniqueName: \"kubernetes.io/empty-dir/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config-out\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086121 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086084    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086146    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086187    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086221    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-2" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:04.086280 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086227    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-secret-prometheus-web-tls-ca\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086302    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086349    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-thanos-prometheus-http-client-file\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086542    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-tls-assets\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-tls-assets\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086594    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086757 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086656    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config\") pod \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\" (UID: \"12ec58dd-c905-47f1-a2f7-66b7a77ffa41\") "
Oct 30 04:16:04.086955 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086935    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.087394 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.086590    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-1" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:04.087532 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.087279    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-0" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:04.088756 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.088730    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-secret-prometheus-web-tls-ca" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.089146 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.089122    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config-out" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:04.089332 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.089283    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-kube-api-access-z8dnb" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "kube-api-access-z8dnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:04.089332 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.089298    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.089501 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.089332    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.089687 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.089667    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-thanos-prometheus-http-client-file" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.089964 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.089933    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.090485 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.090442    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-tls-assets" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:04.093300 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.093280    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:16:04.102553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.102530    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config" pod "12ec58dd-c905-47f1-a2f7-66b7a77ffa41" (UID: "12ec58dd-c905-47f1-a2f7-66b7a77ffa41"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.181535 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.181500    4334 generic.go:356] "Generic (PLEG): container finished" podID="e06a50d5-234c-4074-bd93-bf9462a8083a" containerID="3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a" exitCode=0
Oct 30 04:16:04.181705 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.181552    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"
Oct 30 04:16:04.181705 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.181586    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf" event={"ID":"e06a50d5-234c-4074-bd93-bf9462a8083a","Type":"ContainerDied","Data":"3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a"}
Oct 30 04:16:04.181705 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.181631    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf" event={"ID":"e06a50d5-234c-4074-bd93-bf9462a8083a","Type":"ContainerDied","Data":"bc8ea93b3b517089366dbf77aca2df112062b766741f44b749e148bee05f10a8"}
Oct 30 04:16:04.181705 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.181657    4334 scope.go:122] "RemoveContainer" containerID="3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a"
Oct 30 04:16:04.183553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.183479    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"18f9e9f7-328c-41f8-8377-54c8862e6383","Type":"ContainerStarted","Data":"b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9"}
Oct 30 04:16:04.183553 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.183526    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"18f9e9f7-328c-41f8-8377-54c8862e6383","Type":"ContainerStarted","Data":"a1193bc3b0cd6ec8dc58eefbf2785539b08947297976df1d0675daf185d3133f"}
Oct 30 04:16:04.185277 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.185256    4334 generic.go:356] "Generic (PLEG): container finished" podID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerID="f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6" exitCode=0
Oct 30 04:16:04.185277 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.185274    4334 generic.go:356] "Generic (PLEG): container finished" podID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerID="aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113" exitCode=0
Oct 30 04:16:04.185464 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.185288    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" event={"ID":"3482d570-36ef-4f74-bb80-e6bff98eb3c1","Type":"ContainerDied","Data":"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6"}
Oct 30 04:16:04.185464 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.185318    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"
Oct 30 04:16:04.185464 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.185345    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" event={"ID":"3482d570-36ef-4f74-bb80-e6bff98eb3c1","Type":"ContainerDied","Data":"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113"}
Oct 30 04:16:04.185464 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.185379    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp" event={"ID":"3482d570-36ef-4f74-bb80-e6bff98eb3c1","Type":"ContainerDied","Data":"1cc2a3b5428221dc79b068a79220a3e25dff92e09d7e135c98d1f168be8835aa"}
Oct 30 04:16:04.187284 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187265    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187290    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187304    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187337    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:16:04.187402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187354    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dnb\" (UniqueName: \"kubernetes.io/projected/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-kube-api-access-z8dnb\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187368    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187382    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187399    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187794 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187415    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187794 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187430    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187794 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187470    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.187794 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.187487    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/12ec58dd-c905-47f1-a2f7-66b7a77ffa41-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.188130 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.188106    4334 generic.go:356] "Generic (PLEG): container finished" podID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerID="1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8" exitCode=0
Oct 30 04:16:04.188194 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.188129    4334 generic.go:356] "Generic (PLEG): container finished" podID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerID="3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4" exitCode=0
Oct 30 04:16:04.188194 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.188183    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" event={"ID":"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f","Type":"ContainerDied","Data":"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8"}
Oct 30 04:16:04.188306 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.188228    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" event={"ID":"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f","Type":"ContainerDied","Data":"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4"}
Oct 30 04:16:04.188306 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.188262    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5" event={"ID":"b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f","Type":"ContainerDied","Data":"7d9cb7818ca11e6997b999539d77c08ff3d924d5b37ab76550a383403b9c568e"}
Oct 30 04:16:04.188409 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.188350    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"
Oct 30 04:16:04.192517 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.192282    4334 scope.go:122] "RemoveContainer" containerID="3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a"
Oct 30 04:16:04.192915 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.192886    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a\": container with ID starting with 3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a not found: ID does not exist" containerID="3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a"
Oct 30 04:16:04.193092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.192921    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a"} err="failed to get container status \"3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a\": rpc error: code = NotFound desc = could not find container \"3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a\": container with ID starting with 3db8ac04e40964c574216e77ed0f2e952d58fad5febaf9490bc54a7b211d845a not found: ID does not exist"
Oct 30 04:16:04.193092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.192945    4334 scope.go:122] "RemoveContainer" containerID="f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6"
Oct 30 04:16:04.195666 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.195644    4334 generic.go:356] "Generic (PLEG): container finished" podID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerID="5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4" exitCode=0
Oct 30 04:16:04.195841 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.195735    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ff3a34b7-a892-43fc-a24f-15e575f90960","Type":"ContainerDied","Data":"5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4"}
Oct 30 04:16:04.198576 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.198556    4334 generic.go:356] "Generic (PLEG): container finished" podID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerID="13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b" exitCode=0
Oct 30 04:16:04.198702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.198578    4334 generic.go:356] "Generic (PLEG): container finished" podID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerID="3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5" exitCode=0
Oct 30 04:16:04.198702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.198589    4334 generic.go:356] "Generic (PLEG): container finished" podID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerID="80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6" exitCode=0
Oct 30 04:16:04.198702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.198610    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerDied","Data":"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"}
Oct 30 04:16:04.198702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.198636    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:04.198702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.198660    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerDied","Data":"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"}
Oct 30 04:16:04.199054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.198703    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerDied","Data":"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"}
Oct 30 04:16:04.199054 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.198740    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"12ec58dd-c905-47f1-a2f7-66b7a77ffa41","Type":"ContainerDied","Data":"baa0de2989eb0456a1b592bdbb7497673959fd9c4f130e163554a9b4b73d406c"}
Oct 30 04:16:04.200174 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.200156    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:16:04.204400 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.204378    4334 scope.go:122] "RemoveContainer" containerID="aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113"
Oct 30 04:16:04.213772 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.213757    4334 scope.go:122] "RemoveContainer" containerID="f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6"
Oct 30 04:16:04.214026 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.214011    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6\": container with ID starting with f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6 not found: ID does not exist" containerID="f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6"
Oct 30 04:16:04.214069 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.214032    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6"} err="failed to get container status \"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6\": rpc error: code = NotFound desc = could not find container \"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6\": container with ID starting with f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6 not found: ID does not exist"
Oct 30 04:16:04.214069 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.214046    4334 scope.go:122] "RemoveContainer" containerID="aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113"
Oct 30 04:16:04.214277 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.214259    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113\": container with ID starting with aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113 not found: ID does not exist" containerID="aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113"
Oct 30 04:16:04.214318 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.214283    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113"} err="failed to get container status \"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113\": rpc error: code = NotFound desc = could not find container \"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113\": container with ID starting with aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113 not found: ID does not exist"
Oct 30 04:16:04.214318 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.214302    4334 scope.go:122] "RemoveContainer" containerID="f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6"
Oct 30 04:16:04.214544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.214525    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6"} err="failed to get container status \"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6\": rpc error: code = NotFound desc = could not find container \"f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6\": container with ID starting with f6717763ef139f19d00fc966d235ceb246a99a08d46dc8b42e9e5f0781ea8ea6 not found: ID does not exist"
Oct 30 04:16:04.214592 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.214545    4334 scope.go:122] "RemoveContainer" containerID="aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113"
Oct 30 04:16:04.214734 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.214718    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113"} err="failed to get container status \"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113\": rpc error: code = NotFound desc = could not find container \"aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113\": container with ID starting with aab4823f39398d132489af511d4a8f9378a8f242b2f8da6b2e4cff2314330113 not found: ID does not exist"
Oct 30 04:16:04.214772 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.214735    4334 scope.go:122] "RemoveContainer" containerID="1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8"
Oct 30 04:16:04.222151 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.222131    4334 scope.go:122] "RemoveContainer" containerID="3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4"
Oct 30 04:16:04.229186 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.229169    4334 scope.go:122] "RemoveContainer" containerID="1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8"
Oct 30 04:16:04.229437 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.229419    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8\": container with ID starting with 1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8 not found: ID does not exist" containerID="1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8"
Oct 30 04:16:04.229534 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.229458    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8"} err="failed to get container status \"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8\": rpc error: code = NotFound desc = could not find container \"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8\": container with ID starting with 1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8 not found: ID does not exist"
Oct 30 04:16:04.229534 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.229479    4334 scope.go:122] "RemoveContainer" containerID="3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4"
Oct 30 04:16:04.229724 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.229707    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4\": container with ID starting with 3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4 not found: ID does not exist" containerID="3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4"
Oct 30 04:16:04.229760 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.229729    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4"} err="failed to get container status \"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4\": rpc error: code = NotFound desc = could not find container \"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4\": container with ID starting with 3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4 not found: ID does not exist"
Oct 30 04:16:04.229760 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.229743    4334 scope.go:122] "RemoveContainer" containerID="1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8"
Oct 30 04:16:04.229947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.229928    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8"} err="failed to get container status \"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8\": rpc error: code = NotFound desc = could not find container \"1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8\": container with ID starting with 1a6cdc287fa477d602d7c67af7a343eca9e3451fb9b9e1000707d00fb8ea8dd8 not found: ID does not exist"
Oct 30 04:16:04.230014 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.229948    4334 scope.go:122] "RemoveContainer" containerID="3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4"
Oct 30 04:16:04.230225 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.230196    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4"} err="failed to get container status \"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4\": rpc error: code = NotFound desc = could not find container \"3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4\": container with ID starting with 3115054d0ca091294b24addedadbf8c3c9be38767d4f6711d063d89f26995ba4 not found: ID does not exist"
Oct 30 04:16:04.230265 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.230226    4334 scope.go:122] "RemoveContainer" containerID="13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"
Oct 30 04:16:04.237309 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.237292    4334 scope.go:122] "RemoveContainer" containerID="3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"
Oct 30 04:16:04.244486 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.244469    4334 scope.go:122] "RemoveContainer" containerID="80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"
Oct 30 04:16:04.252964 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.252953    4334 scope.go:122] "RemoveContainer" containerID="2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"
Oct 30 04:16:04.260736 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.260717    4334 scope.go:122] "RemoveContainer" containerID="13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"
Oct 30 04:16:04.260987 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.260968    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b\": container with ID starting with 13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b not found: ID does not exist" containerID="13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"
Oct 30 04:16:04.261029 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.260996    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"} err="failed to get container status \"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b\": rpc error: code = NotFound desc = could not find container \"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b\": container with ID starting with 13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b not found: ID does not exist"
Oct 30 04:16:04.261029 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.261015    4334 scope.go:122] "RemoveContainer" containerID="3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"
Oct 30 04:16:04.261320 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.261299    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5\": container with ID starting with 3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5 not found: ID does not exist" containerID="3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"
Oct 30 04:16:04.261397 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.261327    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"} err="failed to get container status \"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5\": rpc error: code = NotFound desc = could not find container \"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5\": container with ID starting with 3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5 not found: ID does not exist"
Oct 30 04:16:04.261397 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.261345    4334 scope.go:122] "RemoveContainer" containerID="80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"
Oct 30 04:16:04.261715 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.261697    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6\": container with ID starting with 80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6 not found: ID does not exist" containerID="80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"
Oct 30 04:16:04.261767 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.261720    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"} err="failed to get container status \"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6\": rpc error: code = NotFound desc = could not find container \"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6\": container with ID starting with 80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6 not found: ID does not exist"
Oct 30 04:16:04.261767 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.261733    4334 scope.go:122] "RemoveContainer" containerID="2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"
Oct 30 04:16:04.261862 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.261827    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.261816259 podStartE2EDuration="1.261816259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:04.25941408 +0000 UTC m=+5643.266004368" watchObservedRunningTime="2026-10-30 04:16:04.261816259 +0000 UTC m=+5643.268406513"
Oct 30 04:16:04.261993 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:04.261973    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8\": container with ID starting with 2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8 not found: ID does not exist" containerID="2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"
Oct 30 04:16:04.262030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.262001    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"} err="failed to get container status \"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8\": rpc error: code = NotFound desc = could not find container \"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8\": container with ID starting with 2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8 not found: ID does not exist"
Oct 30 04:16:04.262030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.262022    4334 scope.go:122] "RemoveContainer" containerID="13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"
Oct 30 04:16:04.262267 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.262248    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"} err="failed to get container status \"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b\": rpc error: code = NotFound desc = could not find container \"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b\": container with ID starting with 13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b not found: ID does not exist"
Oct 30 04:16:04.262312 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.262268    4334 scope.go:122] "RemoveContainer" containerID="3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"
Oct 30 04:16:04.262516 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.262494    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"} err="failed to get container status \"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5\": rpc error: code = NotFound desc = could not find container \"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5\": container with ID starting with 3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5 not found: ID does not exist"
Oct 30 04:16:04.262590 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.262517    4334 scope.go:122] "RemoveContainer" containerID="80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"
Oct 30 04:16:04.262774 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.262750    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"} err="failed to get container status \"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6\": rpc error: code = NotFound desc = could not find container \"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6\": container with ID starting with 80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6 not found: ID does not exist"
Oct 30 04:16:04.262865 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.262774    4334 scope.go:122] "RemoveContainer" containerID="2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"
Oct 30 04:16:04.263033 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263011    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"} err="failed to get container status \"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8\": rpc error: code = NotFound desc = could not find container \"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8\": container with ID starting with 2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8 not found: ID does not exist"
Oct 30 04:16:04.263088 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263033    4334 scope.go:122] "RemoveContainer" containerID="13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"
Oct 30 04:16:04.263261 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263245    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b"} err="failed to get container status \"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b\": rpc error: code = NotFound desc = could not find container \"13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b\": container with ID starting with 13e2628b937d10afca0b7ff6e5aa011ee1834da7fdf36083e104053a69ec9b9b not found: ID does not exist"
Oct 30 04:16:04.263308 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263261    4334 scope.go:122] "RemoveContainer" containerID="3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"
Oct 30 04:16:04.263481 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263435    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5"} err="failed to get container status \"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5\": rpc error: code = NotFound desc = could not find container \"3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5\": container with ID starting with 3b733b5fcd6844439277e5f4e2d0be6d0d15e3bbd946fec7dcc3dbc8b95064b5 not found: ID does not exist"
Oct 30 04:16:04.263523 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263482    4334 scope.go:122] "RemoveContainer" containerID="80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"
Oct 30 04:16:04.263664 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263647    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6"} err="failed to get container status \"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6\": rpc error: code = NotFound desc = could not find container \"80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6\": container with ID starting with 80592fec5daaaca4e3f2da3a01a2a1d67202ecd1471cb27cdefca2af04ec92d6 not found: ID does not exist"
Oct 30 04:16:04.263704 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263672    4334 scope.go:122] "RemoveContainer" containerID="2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"
Oct 30 04:16:04.263898 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.263876    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8"} err="failed to get container status \"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8\": rpc error: code = NotFound desc = could not find container \"2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8\": container with ID starting with 2c2f726e54ca27d0cd2f2139f3d15a280dd0e9f3c24f65d46b30911f3eb869e8 not found: ID does not exist"
Oct 30 04:16:04.288953 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.288928    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\""
Oct 30 04:16:04.293175 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.293152    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"]
Oct 30 04:16:04.297684 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.297656    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8sppf"]
Oct 30 04:16:04.329725 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.329650    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"]
Oct 30 04:16:04.332962 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.332853    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:04.334657 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.334635    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pd9fp"]
Oct 30 04:16:04.374758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.374724    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:04.381525 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.381491    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:04.413617 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.413584    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"]
Oct 30 04:16:04.418679 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.418633    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2z9z5"]
Oct 30 04:16:04.598458 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.598422    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:04.692429 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692403    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-alertmanager-data-science-monitoringstack-db\") pod \"ff3a34b7-a892-43fc-a24f-15e575f90960\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") "
Oct 30 04:16:04.692596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692464    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-cluster-tls-config\") pod \"ff3a34b7-a892-43fc-a24f-15e575f90960\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") "
Oct 30 04:16:04.692596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692513    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-kube-api-access-ppcjf\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-kube-api-access-ppcjf\") pod \"ff3a34b7-a892-43fc-a24f-15e575f90960\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") "
Oct 30 04:16:04.692596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692542    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-config-volume\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-config-volume\") pod \"ff3a34b7-a892-43fc-a24f-15e575f90960\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") "
Oct 30 04:16:04.692596 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692583    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-web-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-web-config\") pod \"ff3a34b7-a892-43fc-a24f-15e575f90960\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") "
Oct 30 04:16:04.692844 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692658    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-tls-assets\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-tls-assets\") pod \"ff3a34b7-a892-43fc-a24f-15e575f90960\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") "
Oct 30 04:16:04.692844 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692690    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-config-out\") pod \"ff3a34b7-a892-43fc-a24f-15e575f90960\" (UID: \"ff3a34b7-a892-43fc-a24f-15e575f90960\") "
Oct 30 04:16:04.692844 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692773    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-alertmanager-data-science-monitoringstack-db" pod "ff3a34b7-a892-43fc-a24f-15e575f90960" (UID: "ff3a34b7-a892-43fc-a24f-15e575f90960"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:04.693008 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.692964    4334 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.694699 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.694675    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-kube-api-access-ppcjf" pod "ff3a34b7-a892-43fc-a24f-15e575f90960" (UID: "ff3a34b7-a892-43fc-a24f-15e575f90960"). InnerVolumeSpecName "kube-api-access-ppcjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:04.694888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.694867    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-config-out" pod "ff3a34b7-a892-43fc-a24f-15e575f90960" (UID: "ff3a34b7-a892-43fc-a24f-15e575f90960"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:04.695135 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.695108    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-tls-assets" pod "ff3a34b7-a892-43fc-a24f-15e575f90960" (UID: "ff3a34b7-a892-43fc-a24f-15e575f90960"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:04.695206 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.695153    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-config-volume" pod "ff3a34b7-a892-43fc-a24f-15e575f90960" (UID: "ff3a34b7-a892-43fc-a24f-15e575f90960"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.698902 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.698872    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-cluster-tls-config" pod "ff3a34b7-a892-43fc-a24f-15e575f90960" (UID: "ff3a34b7-a892-43fc-a24f-15e575f90960"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.705539 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.705514    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-web-config" pod "ff3a34b7-a892-43fc-a24f-15e575f90960" (UID: "ff3a34b7-a892-43fc-a24f-15e575f90960"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:04.794205 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.794141    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcjf\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-kube-api-access-ppcjf\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.794205 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.794168    4334 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-config-volume\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.794349 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.794206    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.794349 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.794218    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff3a34b7-a892-43fc-a24f-15e575f90960-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.794349 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.794230    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff3a34b7-a892-43fc-a24f-15e575f90960-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:04.794349 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:04.794241    4334 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff3a34b7-a892-43fc-a24f-15e575f90960-cluster-tls-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:05.160140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.160108    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" path="/var/lib/kubelet/pods/12ec58dd-c905-47f1-a2f7-66b7a77ffa41/volumes"
Oct 30 04:16:05.160624 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.160610    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" path="/var/lib/kubelet/pods/3482d570-36ef-4f74-bb80-e6bff98eb3c1/volumes"
Oct 30 04:16:05.161274 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.161260    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a96b75-1434-4c28-a2e1-27be1bef9dae" path="/var/lib/kubelet/pods/57a96b75-1434-4c28-a2e1-27be1bef9dae/volumes"
Oct 30 04:16:05.161636 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.161623    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" path="/var/lib/kubelet/pods/b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f/volumes"
Oct 30 04:16:05.161990 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.161978    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06a50d5-234c-4074-bd93-bf9462a8083a" path="/var/lib/kubelet/pods/e06a50d5-234c-4074-bd93-bf9462a8083a/volumes"
Oct 30 04:16:05.205941 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.205912    4334 generic.go:356] "Generic (PLEG): container finished" podID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerID="a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525" exitCode=0
Oct 30 04:16:05.206053 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.205959    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ff3a34b7-a892-43fc-a24f-15e575f90960","Type":"ContainerDied","Data":"a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525"}
Oct 30 04:16:05.206053 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.205997    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:05.206053 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.206030    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ff3a34b7-a892-43fc-a24f-15e575f90960","Type":"ContainerDied","Data":"ddc523d20860e3dee663efaf8e1adec3d5871f5a31dcbabe056e6637643aa36d"}
Oct 30 04:16:05.206195 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.206069    4334 scope.go:122] "RemoveContainer" containerID="5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4"
Oct 30 04:16:05.210228 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.210205    4334 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-m9vgz\" not found"
Oct 30 04:16:05.214802 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.214785    4334 scope.go:122] "RemoveContainer" containerID="a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525"
Oct 30 04:16:05.221895 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.221875    4334 scope.go:122] "RemoveContainer" containerID="78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8"
Oct 30 04:16:05.229244 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.229228    4334 scope.go:122] "RemoveContainer" containerID="5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4"
Oct 30 04:16:05.229510 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.229490    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4\": container with ID starting with 5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4 not found: ID does not exist" containerID="5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4"
Oct 30 04:16:05.229559 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.229518    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4"} err="failed to get container status \"5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4\": rpc error: code = NotFound desc = could not find container \"5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4\": container with ID starting with 5fbe288d9174a5c28a8b15684d6b60ed75b4b3baeedcd1394b3a7aa7fb307bc4 not found: ID does not exist"
Oct 30 04:16:05.229559 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.229534    4334 scope.go:122] "RemoveContainer" containerID="a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525"
Oct 30 04:16:05.229755 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.229742    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525\": container with ID starting with a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525 not found: ID does not exist" containerID="a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525"
Oct 30 04:16:05.229804 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.229759    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525"} err="failed to get container status \"a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525\": rpc error: code = NotFound desc = could not find container \"a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525\": container with ID starting with a2696897276cb64a90fd36d5f2533a8b0d487101d5ce8826f60abf2b679a4525 not found: ID does not exist"
Oct 30 04:16:05.229804 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.229770    4334 scope.go:122] "RemoveContainer" containerID="78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8"
Oct 30 04:16:05.229977 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.229962    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8\": container with ID starting with 78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8 not found: ID does not exist" containerID="78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8"
Oct 30 04:16:05.230022 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.229980    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8"} err="failed to get container status \"78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8\": rpc error: code = NotFound desc = could not find container \"78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8\": container with ID starting with 78ce4878d227a614426136a1b170a4faedbb24b171757a3248385b602ae8e6d8 not found: ID does not exist"
Oct 30 04:16:05.255994 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.255951    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:05.261098 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:05.261070    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:05.298370 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.298174    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 04:16:05.298370 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.298195    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-tls: secret "data-science-collector-tls" not found
Oct 30 04:16:05.298370 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.298208    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:16:05.298370 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.298249    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal podName:18f9e9f7-328c-41f8-8377-54c8862e6383 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:05.798227295 +0000 UTC m=+5644.804817539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal") pod "data-science-collector-collector-0" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383") : configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 04:16:05.298370 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.298212    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found
Oct 30 04:16:05.298370 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.298273    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs podName:18f9e9f7-328c-41f8-8377-54c8862e6383 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:05.798260229 +0000 UTC m=+5644.804850460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs") pod "data-science-collector-collector-0" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383") : secret "data-science-collector-tls" not found
Oct 30 04:16:05.298370 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.298319    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs podName:18f9e9f7-328c-41f8-8377-54c8862e6383 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:05.798305707 +0000 UTC m=+5644.804895946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383") : secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:16:05.298370 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.298337    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs podName:18f9e9f7-328c-41f8-8377-54c8862e6383 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:05.798327109 +0000 UTC m=+5644.804917342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383") : secret "data-science-collector-monitoring-tls" not found
Oct 30 04:16:05.802830 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.802755    4334 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 04:16:05.802830 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.802775    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found
Oct 30 04:16:05.802830 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.802831    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal podName:18f9e9f7-328c-41f8-8377-54c8862e6383 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:06.802811087 +0000 UTC m=+5645.809401339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal") pod "data-science-collector-collector-0" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383") : configmap "data-science-collector-collector-a0f9182a" not found
Oct 30 04:16:05.803240 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.802841    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-tls: secret "data-science-collector-tls" not found
Oct 30 04:16:05.803240 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.802841    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:16:05.803240 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.802847    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs podName:18f9e9f7-328c-41f8-8377-54c8862e6383 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:06.802839735 +0000 UTC m=+5645.809429967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383") : secret "data-science-collector-monitoring-tls" not found
Oct 30 04:16:05.803240 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.802916    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs podName:18f9e9f7-328c-41f8-8377-54c8862e6383 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:06.80290472 +0000 UTC m=+5645.809494951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs") pod "data-science-collector-collector-0" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383") : secret "data-science-collector-tls" not found
Oct 30 04:16:05.803240 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:05.802931    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs podName:18f9e9f7-328c-41f8-8377-54c8862e6383 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:06.802926048 +0000 UTC m=+5645.809516280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383") : secret "data-science-collector-collector-headless-tls" not found
Oct 30 04:16:06.216675 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.216641    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="18f9e9f7-328c-41f8-8377-54c8862e6383" containerName="otc-container" containerID="cri-o://b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9" gracePeriod=30
Oct 30 04:16:06.465827 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.465803    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:06.609475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.609362    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs\") pod \"18f9e9f7-328c-41f8-8377-54c8862e6383\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") "
Oct 30 04:16:06.609475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.609439    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs\") pod \"18f9e9f7-328c-41f8-8377-54c8862e6383\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") "
Oct 30 04:16:06.609716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.609522    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-data-science-collector-ta-client-cert\") pod \"18f9e9f7-328c-41f8-8377-54c8862e6383\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") "
Oct 30 04:16:06.609716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.609555    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal\" (UniqueName: \"kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal\") pod \"18f9e9f7-328c-41f8-8377-54c8862e6383\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") "
Oct 30 04:16:06.609716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.609588    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f9e9f7-328c-41f8-8377-54c8862e6383-kube-api-access-jhwvq\" (UniqueName: \"kubernetes.io/projected/18f9e9f7-328c-41f8-8377-54c8862e6383-kube-api-access-jhwvq\") pod \"18f9e9f7-328c-41f8-8377-54c8862e6383\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") "
Oct 30 04:16:06.609716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.609617    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs\") pod \"18f9e9f7-328c-41f8-8377-54c8862e6383\" (UID: \"18f9e9f7-328c-41f8-8377-54c8862e6383\") "
Oct 30 04:16:06.610006 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.609978    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal" pod "18f9e9f7-328c-41f8-8377-54c8862e6383" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:06.611853 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.611821    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs" pod "18f9e9f7-328c-41f8-8377-54c8862e6383" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:06.611853 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.611832    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-data-science-collector-ta-client-cert" pod "18f9e9f7-328c-41f8-8377-54c8862e6383" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:06.612008 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.611857    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs" pod "18f9e9f7-328c-41f8-8377-54c8862e6383" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:06.612106 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.612090    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f9e9f7-328c-41f8-8377-54c8862e6383-kube-api-access-jhwvq" pod "18f9e9f7-328c-41f8-8377-54c8862e6383" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383"). InnerVolumeSpecName "kube-api-access-jhwvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:06.612106 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.612099    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs" pod "18f9e9f7-328c-41f8-8377-54c8862e6383" (UID: "18f9e9f7-328c-41f8-8377-54c8862e6383"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:06.711092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.711064    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:06.711092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.711086    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18f9e9f7-328c-41f8-8377-54c8862e6383-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:06.711092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.711095    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhwvq\" (UniqueName: \"kubernetes.io/projected/18f9e9f7-328c-41f8-8377-54c8862e6383-kube-api-access-jhwvq\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:06.711305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.711105    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:06.711305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.711113    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:06.711305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:06.711123    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18f9e9f7-328c-41f8-8377-54c8862e6383-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:07.159702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.159673    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" path="/var/lib/kubelet/pods/ff3a34b7-a892-43fc-a24f-15e575f90960/volumes"
Oct 30 04:16:07.221841 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.221817    4334 generic.go:356] "Generic (PLEG): container finished" podID="18f9e9f7-328c-41f8-8377-54c8862e6383" containerID="b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9" exitCode=0
Oct 30 04:16:07.221979 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.221868    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:07.221979 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.221905    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"18f9e9f7-328c-41f8-8377-54c8862e6383","Type":"ContainerDied","Data":"b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9"}
Oct 30 04:16:07.221979 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.221953    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"18f9e9f7-328c-41f8-8377-54c8862e6383","Type":"ContainerDied","Data":"a1193bc3b0cd6ec8dc58eefbf2785539b08947297976df1d0675daf185d3133f"}
Oct 30 04:16:07.221979 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.221975    4334 scope.go:122] "RemoveContainer" containerID="b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9"
Oct 30 04:16:07.230187 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.230171    4334 scope.go:122] "RemoveContainer" containerID="b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9"
Oct 30 04:16:07.230430 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:07.230409    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9\": container with ID starting with b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9 not found: ID does not exist" containerID="b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9"
Oct 30 04:16:07.230518 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.230440    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9"} err="failed to get container status \"b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9\": rpc error: code = NotFound desc = could not find container \"b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9\": container with ID starting with b30b82845222aff4cdd33c54d5ee38dd51979fab01fbefaa7c60cd658fb790e9 not found: ID does not exist"
Oct 30 04:16:07.270365 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.270340    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:07.281482 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:07.281457    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:09.158003 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:09.157969    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f9e9f7-328c-41f8-8377-54c8862e6383" path="/var/lib/kubelet/pods/18f9e9f7-328c-41f8-8377-54c8862e6383/volumes"
Oct 30 04:16:10.047359 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047330    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"]
Oct 30 04:16:10.047699 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047683    4334 cpu_manager.go:414] "containerMap: removing container" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="init-config-reloader"
Oct 30 04:16:10.047756 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047702    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="init-config-reloader"
Oct 30 04:16:10.047756 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047729    4334 cpu_manager.go:414] "containerMap: removing container" podUID="18f9e9f7-328c-41f8-8377-54c8862e6383" containerName="otc-container"
Oct 30 04:16:10.047756 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047736    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f9e9f7-328c-41f8-8377-54c8862e6383" containerName="otc-container"
Oct 30 04:16:10.047756 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047750    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerName="kube-rbac-proxy"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047758    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerName="kube-rbac-proxy"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047768    4334 cpu_manager.go:414] "containerMap: removing container" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="config-reloader"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047773    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="config-reloader"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047781    4334 cpu_manager.go:414] "containerMap: removing container" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerName="prom-label-proxy"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047785    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerName="prom-label-proxy"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047793    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="init-config-reloader"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047797    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="init-config-reloader"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047804    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerName="kube-rbac-proxy"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047808    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerName="kube-rbac-proxy"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047814    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerName="prom-label-proxy"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047818    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerName="prom-label-proxy"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047826    4334 cpu_manager.go:414] "containerMap: removing container" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="prometheus"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047830    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="prometheus"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047835    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="config-reloader"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047839    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="config-reloader"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047847    4334 cpu_manager.go:414] "containerMap: removing container" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="alertmanager"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047875    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="alertmanager"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047882    4334 cpu_manager.go:414] "containerMap: removing container" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="thanos-sidecar"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047887    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="thanos-sidecar"
Oct 30 04:16:10.047888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047896    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e06a50d5-234c-4074-bd93-bf9462a8083a" containerName="kube-rbac-proxy"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047910    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06a50d5-234c-4074-bd93-bf9462a8083a" containerName="kube-rbac-proxy"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.047985    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06a50d5-234c-4074-bd93-bf9462a8083a" containerName="kube-rbac-proxy"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048000    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f9e9f7-328c-41f8-8377-54c8862e6383" containerName="otc-container"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048008    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerName="kube-rbac-proxy"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048016    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="config-reloader"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048029    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f6dbf0-96a6-43a9-a4c1-bdfe011cec7f" containerName="prom-label-proxy"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048038    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerName="kube-rbac-proxy"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048047    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="3482d570-36ef-4f74-bb80-e6bff98eb3c1" containerName="prom-label-proxy"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048056    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="prometheus"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048062    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="config-reloader"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048070    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3a34b7-a892-43fc-a24f-15e575f90960" containerName="alertmanager"
Oct 30 04:16:10.048536 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.048077    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ec58dd-c905-47f1-a2f7-66b7a77ffa41" containerName="thanos-sidecar"
Oct 30 04:16:10.052638 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.052610    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.067129 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.067105    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"]
Oct 30 04:16:10.068421 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.068400    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\""
Oct 30 04:16:10.076400 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.076380    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:16:10.076640 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.076612    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:16:10.076739 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.076689    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:16:10.079350 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.079330    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-dtrcn\""
Oct 30 04:16:10.079350 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.079342    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\""
Oct 30 04:16:10.079538 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.079432    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:16:10.085000 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.084980    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"]
Oct 30 04:16:10.088357 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.088343    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.106738 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.106717    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"]
Oct 30 04:16:10.124425 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.124402    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"]
Oct 30 04:16:10.127845 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.127829    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.139399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.139380    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.139528 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.139413    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.139528 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.139433    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.139624 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.139546    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.139624 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.139574    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bp8\" (UniqueName: \"kubernetes.io/projected/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-api-access-d8bp8\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.146293 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.146268    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"]
Oct 30 04:16:10.152600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.152577    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-696mz\""
Oct 30 04:16:10.152694 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.152673    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 04:16:10.152739 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.152722    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-sbc2v\""
Oct 30 04:16:10.152937 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.152825    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 04:16:10.168674 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.168649    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 04:16:10.168991 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.168703    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 04:16:10.168991 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.168617    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 04:16:10.240336 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240308    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f566a17f-43cf-481d-9732-2ebc9bc71391-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.240531 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240345    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.240531 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240363    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.240531 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240388    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtmb\" (UniqueName: \"kubernetes.io/projected/6f84781d-58a8-422d-a90f-60692f2ad4f6-kube-api-access-tvtmb\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.240531 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:10.240512    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found
Oct 30 04:16:10.240767 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240557    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.240767 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:10.240568    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert podName:734f8b87-0bbd-4961-8cbd-867fe23f2244 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:10.740551856 +0000 UTC m=+5649.747142105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-v6lvk" (UID: "734f8b87-0bbd-4961-8cbd-867fe23f2244") : secret "prometheus-operated-tls" not found
Oct 30 04:16:10.240767 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240605    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrh9x\" (UniqueName: \"kubernetes.io/projected/f566a17f-43cf-481d-9732-2ebc9bc71391-kube-api-access-lrh9x\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.240767 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240650    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bp8\" (UniqueName: \"kubernetes.io/projected/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-api-access-d8bp8\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.240767 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240692    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.240993 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240809    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.240993 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240889    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.240993 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.240930    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6f84781d-58a8-422d-a90f-60692f2ad4f6-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.241117 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.241016    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f566a17f-43cf-481d-9732-2ebc9bc71391-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.241117 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.241046    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.241306 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.241284    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.242825 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.242804    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.243233 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.243217    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.259719 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.259698    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bp8\" (UniqueName: \"kubernetes.io/projected/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-api-access-d8bp8\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.342373 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.342305    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.342373 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.342338    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.342373 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.342357    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6f84781d-58a8-422d-a90f-60692f2ad4f6-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.342692 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.342380    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f566a17f-43cf-481d-9732-2ebc9bc71391-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.342692 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.342403    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f566a17f-43cf-481d-9732-2ebc9bc71391-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.342692 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.342484    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtmb\" (UniqueName: \"kubernetes.io/projected/6f84781d-58a8-422d-a90f-60692f2ad4f6-kube-api-access-tvtmb\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.342894 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.342781    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrh9x\" (UniqueName: \"kubernetes.io/projected/f566a17f-43cf-481d-9732-2ebc9bc71391-kube-api-access-lrh9x\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.342894 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.342824    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.343153 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.343130    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.343232 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.343169    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f566a17f-43cf-481d-9732-2ebc9bc71391-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.343232 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.343221    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.343321 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.343279    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.344896 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.344873    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6f84781d-58a8-422d-a90f-60692f2ad4f6-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.345010 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.344988    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f566a17f-43cf-481d-9732-2ebc9bc71391-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.360357 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.360334    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrh9x\" (UniqueName: \"kubernetes.io/projected/f566a17f-43cf-481d-9732-2ebc9bc71391-kube-api-access-lrh9x\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-jbrf6\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.360511 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.360492    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtmb\" (UniqueName: \"kubernetes.io/projected/6f84781d-58a8-422d-a90f-60692f2ad4f6-kube-api-access-tvtmb\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-drwv2\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.404139 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.404115    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:10.442371 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.442220    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:10.701495 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.701459    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"]
Oct 30 04:16:10.705324 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:16:10.705285    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f84781d_58a8_422d_a90f_60692f2ad4f6.slice/crio-fbc63ac3bcd04278cc686d651a9c86724524656f5df40a2faac58e9d9676cf8f WatchSource:0}: Error finding container fbc63ac3bcd04278cc686d651a9c86724524656f5df40a2faac58e9d9676cf8f: Status 404 returned error can't find the container with id fbc63ac3bcd04278cc686d651a9c86724524656f5df40a2faac58e9d9676cf8f
Oct 30 04:16:10.747298 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.747273    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.749037 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.749013    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"]
Oct 30 04:16:10.749861 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.749840    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-v6lvk\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:10.752726 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:16:10.752697    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf566a17f_43cf_481d_9732_2ebc9bc71391.slice/crio-16c7291234650194f6486e30ec6b3d13008b9119191fab7b12e60053f1f0795c WatchSource:0}: Error finding container 16c7291234650194f6486e30ec6b3d13008b9119191fab7b12e60053f1f0795c: Status 404 returned error can't find the container with id 16c7291234650194f6486e30ec6b3d13008b9119191fab7b12e60053f1f0795c
Oct 30 04:16:10.965323 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:10.965241    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:11.242563 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.242481    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" event={"ID":"f566a17f-43cf-481d-9732-2ebc9bc71391","Type":"ContainerStarted","Data":"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8"}
Oct 30 04:16:11.242563 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.242544    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" event={"ID":"f566a17f-43cf-481d-9732-2ebc9bc71391","Type":"ContainerStarted","Data":"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9"}
Oct 30 04:16:11.243026 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.242579    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" event={"ID":"f566a17f-43cf-481d-9732-2ebc9bc71391","Type":"ContainerStarted","Data":"16c7291234650194f6486e30ec6b3d13008b9119191fab7b12e60053f1f0795c"}
Oct 30 04:16:11.244414 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.244383    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" event={"ID":"6f84781d-58a8-422d-a90f-60692f2ad4f6","Type":"ContainerStarted","Data":"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c"}
Oct 30 04:16:11.244551 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.244461    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" event={"ID":"6f84781d-58a8-422d-a90f-60692f2ad4f6","Type":"ContainerStarted","Data":"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03"}
Oct 30 04:16:11.244551 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.244495    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" event={"ID":"6f84781d-58a8-422d-a90f-60692f2ad4f6","Type":"ContainerStarted","Data":"fbc63ac3bcd04278cc686d651a9c86724524656f5df40a2faac58e9d9676cf8f"}
Oct 30 04:16:11.267466 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.267328    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"]
Oct 30 04:16:11.270076 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:16:11.270052    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734f8b87_0bbd_4961_8cbd_867fe23f2244.slice/crio-715e9a3d17e2f70ad1d3dca24f6c817c2c8a9f5f02cd1d94d9694f8a2f0fa7c0 WatchSource:0}: Error finding container 715e9a3d17e2f70ad1d3dca24f6c817c2c8a9f5f02cd1d94d9694f8a2f0fa7c0: Status 404 returned error can't find the container with id 715e9a3d17e2f70ad1d3dca24f6c817c2c8a9f5f02cd1d94d9694f8a2f0fa7c0
Oct 30 04:16:11.284518 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.284436    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" podStartSLOduration=1.284417514 podStartE2EDuration="1.284417514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:11.28216784 +0000 UTC m=+5650.288758095" watchObservedRunningTime="2026-10-30 04:16:11.284417514 +0000 UTC m=+5650.291007769"
Oct 30 04:16:11.320905 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.320855    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" podStartSLOduration=1.32084131 podStartE2EDuration="1.32084131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:11.319327184 +0000 UTC m=+5650.325917442" watchObservedRunningTime="2026-10-30 04:16:11.32084131 +0000 UTC m=+5650.327431564"
Oct 30 04:16:11.528040 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.527965    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:11.532837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.532808    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.549585 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.548718    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:16:11.549585 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.548749    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:16:11.549585 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.548824    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:16:11.549585 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.548858    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:16:11.549585 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.548964    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:16:11.551125 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.551102    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:11.655522 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.655490    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.655690 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.655534    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzwn\" (UniqueName: \"kubernetes.io/projected/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-kube-api-access-7qzwn\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.655690 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.655596    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.655690 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.655633    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.655690 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.655656    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.655846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.655777    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.756617 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.756571    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.756617 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.756626    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.756896 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.756693    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.756896 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.756738    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.756896 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.756771    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzwn\" (UniqueName: \"kubernetes.io/projected/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-kube-api-access-7qzwn\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.756896 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.756797    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.757557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.757529    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.759503 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.759476    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.759503 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.759500    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.759934 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.759912    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.759993 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.759918    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.774829 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.774808    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzwn\" (UniqueName: \"kubernetes.io/projected/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-kube-api-access-7qzwn\") pod \"data-science-collector-collector-0\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:11.846663 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:11.846604    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:12.143126 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.143053    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:12.148091 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:16:12.148066    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe50238_d7ff_4ba4_bea7_92658aa9cd3d.slice/crio-a0eab5246c8af034cd25c10d4d86d1f6e935a53594cad4127f851918e25e53ce WatchSource:0}: Error finding container a0eab5246c8af034cd25c10d4d86d1f6e935a53594cad4127f851918e25e53ce: Status 404 returned error can't find the container with id a0eab5246c8af034cd25c10d4d86d1f6e935a53594cad4127f851918e25e53ce
Oct 30 04:16:12.150185 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.150151    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:16:12.150265 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.150216    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:16:12.250991 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.250961    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk" event={"ID":"734f8b87-0bbd-4961-8cbd-867fe23f2244","Type":"ContainerStarted","Data":"c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb"}
Oct 30 04:16:12.251417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.251025    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:12.251417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.251051    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk" event={"ID":"734f8b87-0bbd-4961-8cbd-867fe23f2244","Type":"ContainerStarted","Data":"715e9a3d17e2f70ad1d3dca24f6c817c2c8a9f5f02cd1d94d9694f8a2f0fa7c0"}
Oct 30 04:16:12.252702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.252673    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"abe50238-d7ff-4ba4-bea7-92658aa9cd3d","Type":"ContainerStarted","Data":"f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766"}
Oct 30 04:16:12.252821 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.252718    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"abe50238-d7ff-4ba4-bea7-92658aa9cd3d","Type":"ContainerStarted","Data":"a0eab5246c8af034cd25c10d4d86d1f6e935a53594cad4127f851918e25e53ce"}
Oct 30 04:16:12.287222 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.287171    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk" podStartSLOduration=2.28715779 podStartE2EDuration="2.28715779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:12.286400411 +0000 UTC m=+5651.292990665" watchObservedRunningTime="2026-10-30 04:16:12.28715779 +0000 UTC m=+5651.293748043"
Oct 30 04:16:12.322832 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:12.322789    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.322772671 podStartE2EDuration="1.322772671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:12.321851548 +0000 UTC m=+5651.328441813" watchObservedRunningTime="2026-10-30 04:16:12.322772671 +0000 UTC m=+5651.329362972"
Oct 30 04:16:15.109809 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.109774    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:15.113691 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.113669    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.143391 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.143352    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:16:15.143543 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.143424    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:16:15.143543 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.143435    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:16:15.143543 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.143512    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:16:15.143684 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.143572    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-nnncs\""
Oct 30 04:16:15.145073 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.145055    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:15.290632 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.290597    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.290632 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.290638    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.290905 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.290676    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.290905 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.290888    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5xh\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-kube-api-access-kg5xh\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.291033 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.290988    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.291096 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.291043    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.291096 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.291085    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.381408 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.381334    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:15.385247 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.385231    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.391559 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.391532    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.391698 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.391563    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.391698 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.391602    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.391698 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.391631    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5xh\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-kube-api-access-kg5xh\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.391698 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.391667    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.391883 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.391703    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.391883 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.391751    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.392412 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.392381    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.394547 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.394503    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.394856 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.394832    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.394938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.394914    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.395951 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.395924    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.396149 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.396129    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.406824 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.406796    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:15.415202 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.415178    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5xh\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-kube-api-access-kg5xh\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.432133 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.432114    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:15.470370 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470342    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:16:15.470557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470419    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:16:15.470557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470355    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:16:15.470557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470501    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:16:15.470557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470554    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:16:15.470764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470628    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:16:15.470764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470654    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:16:15.470764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470660    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:16:15.470764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.470698    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-42x42\""
Oct 30 04:16:15.498599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.498218    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.498599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.498272    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.498599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.498303    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f5430e9-3440-4a11-b4bd-68583ca2864d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.498599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.498341    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.498599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.498371    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.498599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.498395    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.498599 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.498428    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7px89\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-kube-api-access-7px89\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.499597 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.499573    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.499729 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.499714    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.499809 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.499747    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.499874 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.499861    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.499938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.499900    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.600520 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600490    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.600520 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600524    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600568    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600596    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600627    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600658    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600686    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f5430e9-3440-4a11-b4bd-68583ca2864d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600717    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600743    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600768    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600799    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7px89\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-kube-api-access-7px89\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601051 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.600839    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601612 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.601236    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.601612 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.601289    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.602622 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.602589    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.603604 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.603288    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.603716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.603671    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.603716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.603671    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.603848 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.603816    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f5430e9-3440-4a11-b4bd-68583ca2864d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.603963 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.603945    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.604180 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.604154    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.604598 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.604581    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.604973 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.604957    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.621004 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.620985    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7px89\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-kube-api-access-7px89\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:15.744010 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:15.743979    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:15.747812 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:16:15.747785    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25bc8f4_d802_4013_8912_57b3149d79c4.slice/crio-a4a72e787e8b89693cbc105fbedaed17d6cdf147cd0e489be966c9358d2eb8cf WatchSource:0}: Error finding container a4a72e787e8b89693cbc105fbedaed17d6cdf147cd0e489be966c9358d2eb8cf: Status 404 returned error can't find the container with id a4a72e787e8b89693cbc105fbedaed17d6cdf147cd0e489be966c9358d2eb8cf
Oct 30 04:16:16.273606 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:16.273572    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c25bc8f4-d802-4013-8912-57b3149d79c4","Type":"ContainerStarted","Data":"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6"}
Oct 30 04:16:16.274014 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:16.273628    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c25bc8f4-d802-4013-8912-57b3149d79c4","Type":"ContainerStarted","Data":"a4a72e787e8b89693cbc105fbedaed17d6cdf147cd0e489be966c9358d2eb8cf"}
Oct 30 04:16:18.259909 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:18.259879    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:19.536995 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:19.536971    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:19.563611 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:19.563581    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:19.638190 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:19.638162    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:19.638314 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:19.638261    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:19.651835 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:19.651812    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:19.751176 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:19.751151    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:19.775887 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:19.775865    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:19.927337 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:19.927305    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:20.258316 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:16:20.258289    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5430e9_3440_4a11_b4bd_68583ca2864d.slice/crio-80f5014ecdcbf3d10c56fa5b88739361e37641abf55fe05b167d2964356d614d WatchSource:0}: Error finding container 80f5014ecdcbf3d10c56fa5b88739361e37641abf55fe05b167d2964356d614d: Status 404 returned error can't find the container with id 80f5014ecdcbf3d10c56fa5b88739361e37641abf55fe05b167d2964356d614d
Oct 30 04:16:20.265853 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:20.265738    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:20.293785 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:20.293761    4334 generic.go:356] "Generic (PLEG): container finished" podID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerID="61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6" exitCode=0
Oct 30 04:16:20.293903 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:20.293853    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c25bc8f4-d802-4013-8912-57b3149d79c4","Type":"ContainerDied","Data":"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6"}
Oct 30 04:16:20.295220 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:20.295194    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerStarted","Data":"80f5014ecdcbf3d10c56fa5b88739361e37641abf55fe05b167d2964356d614d"}
Oct 30 04:16:20.977541 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:20.977440    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:16:21.302176 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:21.302088    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c25bc8f4-d802-4013-8912-57b3149d79c4","Type":"ContainerStarted","Data":"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b"}
Oct 30 04:16:21.302176 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:21.302155    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c25bc8f4-d802-4013-8912-57b3149d79c4","Type":"ContainerStarted","Data":"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11"}
Oct 30 04:16:21.302557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:21.302529    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:21.303706 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:21.303683    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerStarted","Data":"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"}
Oct 30 04:16:21.362183 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:21.362140    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.978077803 podStartE2EDuration="6.362127973s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.38405017s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:21.360945513 +0000 UTC m=+5660.367535767" watchObservedRunningTime="2026-10-30 04:16:21.362127973 +0000 UTC m=+5660.368718261"
Oct 30 04:16:24.319666 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:24.319636    4334 generic.go:356] "Generic (PLEG): container finished" podID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerID="e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390" exitCode=0
Oct 30 04:16:24.320144 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:24.319714    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerDied","Data":"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"}
Oct 30 04:16:25.326626 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:25.326593    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerStarted","Data":"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"}
Oct 30 04:16:25.326992 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:25.326652    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerStarted","Data":"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"}
Oct 30 04:16:25.326992 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:25.326675    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerStarted","Data":"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"}
Oct 30 04:16:25.396113 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:25.396067    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=7.005821312 podStartE2EDuration="10.396054671s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.390233359s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:25.391840221 +0000 UTC m=+5664.398430475" watchObservedRunningTime="2026-10-30 04:16:25.396054671 +0000 UTC m=+5664.402644954"
Oct 30 04:16:27.310438 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:27.310410    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:29.927718 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:29.927686    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:32.638391 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:32.638358    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:32.638956 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:32.638628    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="abe50238-d7ff-4ba4-bea7-92658aa9cd3d" containerName="otc-container" containerID="cri-o://f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766" gracePeriod=30
Oct 30 04:16:32.893797 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:32.893741    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.061763 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.061732    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otc-internal\") pod \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") "
Oct 30 04:16:33.061940 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.061771    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-monitoring-tls-certs\") pod \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") "
Oct 30 04:16:33.061940 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.061798    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otlp-tls-certs\") pod \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") "
Oct 30 04:16:33.061940 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.061845    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-data-science-collector-ta-client-cert\") pod \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") "
Oct 30 04:16:33.061940 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.061894    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-tls-certs\") pod \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") "
Oct 30 04:16:33.061940 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.061937    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-kube-api-access-7qzwn\" (UniqueName: \"kubernetes.io/projected/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-kube-api-access-7qzwn\") pod \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\" (UID: \"abe50238-d7ff-4ba4-bea7-92658aa9cd3d\") "
Oct 30 04:16:33.062213 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.062173    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otc-internal" pod "abe50238-d7ff-4ba4-bea7-92658aa9cd3d" (UID: "abe50238-d7ff-4ba4-bea7-92658aa9cd3d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:33.064055 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.064020    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otlp-tls-certs" pod "abe50238-d7ff-4ba4-bea7-92658aa9cd3d" (UID: "abe50238-d7ff-4ba4-bea7-92658aa9cd3d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:33.064055 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.064025    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-kube-api-access-7qzwn" pod "abe50238-d7ff-4ba4-bea7-92658aa9cd3d" (UID: "abe50238-d7ff-4ba4-bea7-92658aa9cd3d"). InnerVolumeSpecName "kube-api-access-7qzwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:33.064233 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.064104    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-data-science-collector-ta-client-cert" pod "abe50238-d7ff-4ba4-bea7-92658aa9cd3d" (UID: "abe50238-d7ff-4ba4-bea7-92658aa9cd3d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:33.064337 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.064320    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-tls-certs" pod "abe50238-d7ff-4ba4-bea7-92658aa9cd3d" (UID: "abe50238-d7ff-4ba4-bea7-92658aa9cd3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:33.064374 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.064331    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-monitoring-tls-certs" pod "abe50238-d7ff-4ba4-bea7-92658aa9cd3d" (UID: "abe50238-d7ff-4ba4-bea7-92658aa9cd3d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:33.163412 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.163381    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:33.163412 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.163408    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qzwn\" (UniqueName: \"kubernetes.io/projected/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-kube-api-access-7qzwn\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:33.163610 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.163423    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:33.163610 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.163436    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:33.163610 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.163469    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:33.163610 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.163484    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abe50238-d7ff-4ba4-bea7-92658aa9cd3d-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:33.368369 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.368339    4334 generic.go:356] "Generic (PLEG): container finished" podID="abe50238-d7ff-4ba4-bea7-92658aa9cd3d" containerID="f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766" exitCode=0
Oct 30 04:16:33.368540 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.368401    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.368540 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.368405    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"abe50238-d7ff-4ba4-bea7-92658aa9cd3d","Type":"ContainerDied","Data":"f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766"}
Oct 30 04:16:33.368540 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.368473    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"abe50238-d7ff-4ba4-bea7-92658aa9cd3d","Type":"ContainerDied","Data":"a0eab5246c8af034cd25c10d4d86d1f6e935a53594cad4127f851918e25e53ce"}
Oct 30 04:16:33.368540 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.368505    4334 scope.go:122] "RemoveContainer" containerID="f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766"
Oct 30 04:16:33.377111 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.377089    4334 scope.go:122] "RemoveContainer" containerID="f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766"
Oct 30 04:16:33.377377 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:33.377357    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766\": container with ID starting with f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766 not found: ID does not exist" containerID="f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766"
Oct 30 04:16:33.377430 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.377385    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766"} err="failed to get container status \"f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766\": rpc error: code = NotFound desc = could not find container \"f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766\": container with ID starting with f52d82b8e3e1baa04d54899f3b232bf1a8825ed5f993fa4830240a908c78f766 not found: ID does not exist"
Oct 30 04:16:33.415180 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.415123    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:33.426115 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.426091    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:33.451825 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.451804    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:33.452190 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.452165    4334 cpu_manager.go:414] "containerMap: removing container" podUID="abe50238-d7ff-4ba4-bea7-92658aa9cd3d" containerName="otc-container"
Oct 30 04:16:33.452190 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.452180    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe50238-d7ff-4ba4-bea7-92658aa9cd3d" containerName="otc-container"
Oct 30 04:16:33.452335 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.452260    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe50238-d7ff-4ba4-bea7-92658aa9cd3d" containerName="otc-container"
Oct 30 04:16:33.456554 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.456534    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.471274 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.471220    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:16:33.471442 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.471419    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:16:33.471603 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.471579    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:16:33.471752 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.471730    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8rxn8\""
Oct 30 04:16:33.472791 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.472771    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:33.481646 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.481613    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:16:33.481751 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.481649    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:16:33.565354 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.565327    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.565354 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.565356    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.565538 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.565473    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/49777d61-2a10-4d15-a980-d5ce6618b44d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.565538 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.565523    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.565620 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.565604    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9448x\" (UniqueName: \"kubernetes.io/projected/49777d61-2a10-4d15-a980-d5ce6618b44d-kube-api-access-9448x\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.565659 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.565634    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.666675 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.666606    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.666675 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.666644    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9448x\" (UniqueName: \"kubernetes.io/projected/49777d61-2a10-4d15-a980-d5ce6618b44d-kube-api-access-9448x\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.666675 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.666671    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.667208 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.666713    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.667208 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.666737    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.667208 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.666819    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/49777d61-2a10-4d15-a980-d5ce6618b44d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.667600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.667578    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/49777d61-2a10-4d15-a980-d5ce6618b44d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.669300 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.669277    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.669604 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.669584    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.669652 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.669607    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.669689 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.669678    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.683882 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.683863    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9448x\" (UniqueName: \"kubernetes.io/projected/49777d61-2a10-4d15-a980-d5ce6618b44d-kube-api-access-9448x\") pod \"data-science-collector-collector-0\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:33.768758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:33.768732    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:34.063597 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:34.063520    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:34.067437 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:16:34.067409    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49777d61_2a10_4d15_a980_d5ce6618b44d.slice/crio-7544579b6cf5b54938c52252f95dc590f916988aaf2a6bd335849ea6b7b4f2c7 WatchSource:0}: Error finding container 7544579b6cf5b54938c52252f95dc590f916988aaf2a6bd335849ea6b7b4f2c7: Status 404 returned error can't find the container with id 7544579b6cf5b54938c52252f95dc590f916988aaf2a6bd335849ea6b7b4f2c7
Oct 30 04:16:34.069661 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:34.069627    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:16:34.069769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:34.069706    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:16:34.376403 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:34.376329    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"49777d61-2a10-4d15-a980-d5ce6618b44d","Type":"ContainerStarted","Data":"5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b"}
Oct 30 04:16:34.376403 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:34.376379    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"49777d61-2a10-4d15-a980-d5ce6618b44d","Type":"ContainerStarted","Data":"7544579b6cf5b54938c52252f95dc590f916988aaf2a6bd335849ea6b7b4f2c7"}
Oct 30 04:16:34.414860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:34.414797    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.414780403 podStartE2EDuration="1.414780403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:34.412916376 +0000 UTC m=+5673.419506680" watchObservedRunningTime="2026-10-30 04:16:34.414780403 +0000 UTC m=+5673.421370659"
Oct 30 04:16:34.927688 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:34.927660    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:34.933101 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:34.933081    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:35.159325 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:35.159294    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe50238-d7ff-4ba4-bea7-92658aa9cd3d" path="/var/lib/kubelet/pods/abe50238-d7ff-4ba4-bea7-92658aa9cd3d/volumes"
Oct 30 04:16:35.385786 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:35.385723    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:50.978689 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:50.978650    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:16:51.730621 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.730577    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:51.730838 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.730807    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="49777d61-2a10-4d15-a980-d5ce6618b44d" containerName="otc-container" containerID="cri-o://5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b" gracePeriod=30
Oct 30 04:16:51.938420 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.938378    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"]
Oct 30 04:16:51.938653 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.938621    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk" podUID="734f8b87-0bbd-4961-8cbd-867fe23f2244" containerName="kube-rbac-proxy" containerID="cri-o://c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb" gracePeriod=30
Oct 30 04:16:51.954475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.954437    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"]
Oct 30 04:16:51.954741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.954688    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerName="prom-label-proxy" containerID="cri-o://0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c" gracePeriod=30
Oct 30 04:16:51.954880 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.954678    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerName="kube-rbac-proxy" containerID="cri-o://4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03" gracePeriod=30
Oct 30 04:16:51.976532 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.976504    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"]
Oct 30 04:16:51.976738 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.976719    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerName="kube-rbac-proxy" containerID="cri-o://439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9" gracePeriod=30
Oct 30 04:16:51.976804 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:51.976748    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerName="prom-label-proxy" containerID="cri-o://bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8" gracePeriod=30
Oct 30 04:16:52.219699 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.219667    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:52.259702 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.259521    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:52.271988 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.271958    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:52.335484 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.335426    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-ca\") pod \"734f8b87-0bbd-4961-8cbd-867fe23f2244\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") "
Oct 30 04:16:52.335683 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.335492    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f566a17f-43cf-481d-9732-2ebc9bc71391-tls\" (UniqueName: \"kubernetes.io/secret/f566a17f-43cf-481d-9732-2ebc9bc71391-tls\") pod \"f566a17f-43cf-481d-9732-2ebc9bc71391\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") "
Oct 30 04:16:52.335683 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.335521    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f566a17f-43cf-481d-9732-2ebc9bc71391-config\" (UniqueName: \"kubernetes.io/configmap/f566a17f-43cf-481d-9732-2ebc9bc71391-config\") pod \"f566a17f-43cf-481d-9732-2ebc9bc71391\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") "
Oct 30 04:16:52.335809 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.335787    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-rbac-proxy-config\") pod \"734f8b87-0bbd-4961-8cbd-867fe23f2244\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") "
Oct 30 04:16:52.335888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.335843    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f84781d-58a8-422d-a90f-60692f2ad4f6-kube-api-access-tvtmb\" (UniqueName: \"kubernetes.io/projected/6f84781d-58a8-422d-a90f-60692f2ad4f6-kube-api-access-tvtmb\") pod \"6f84781d-58a8-422d-a90f-60692f2ad4f6\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") "
Oct 30 04:16:52.335888 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.335867    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-config\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-config\") pod \"6f84781d-58a8-422d-a90f-60692f2ad4f6\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") "
Oct 30 04:16:52.336003 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.335906    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f566a17f-43cf-481d-9732-2ebc9bc71391-kube-api-access-lrh9x\" (UniqueName: \"kubernetes.io/projected/f566a17f-43cf-481d-9732-2ebc9bc71391-kube-api-access-lrh9x\") pod \"f566a17f-43cf-481d-9732-2ebc9bc71391\" (UID: \"f566a17f-43cf-481d-9732-2ebc9bc71391\") "
Oct 30 04:16:52.336003 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.335962    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-client-ca\") pod \"6f84781d-58a8-422d-a90f-60692f2ad4f6\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") "
Oct 30 04:16:52.336107 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.336013    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-tls\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-tls\") pod \"734f8b87-0bbd-4961-8cbd-867fe23f2244\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") "
Oct 30 04:16:52.336107 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.336077    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert\") pod \"734f8b87-0bbd-4961-8cbd-867fe23f2244\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") "
Oct 30 04:16:52.336220 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.336118    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-api-access-d8bp8\" (UniqueName: \"kubernetes.io/projected/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-api-access-d8bp8\") pod \"734f8b87-0bbd-4961-8cbd-867fe23f2244\" (UID: \"734f8b87-0bbd-4961-8cbd-867fe23f2244\") "
Oct 30 04:16:52.336220 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.336144    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-prometheus-tls-ca\") pod \"6f84781d-58a8-422d-a90f-60692f2ad4f6\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") "
Oct 30 04:16:52.336220 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.336174    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f84781d-58a8-422d-a90f-60692f2ad4f6-tls\" (UniqueName: \"kubernetes.io/secret/6f84781d-58a8-422d-a90f-60692f2ad4f6-tls\") pod \"6f84781d-58a8-422d-a90f-60692f2ad4f6\" (UID: \"6f84781d-58a8-422d-a90f-60692f2ad4f6\") "
Oct 30 04:16:52.337311 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.337280    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f566a17f-43cf-481d-9732-2ebc9bc71391-config" pod "f566a17f-43cf-481d-9732-2ebc9bc71391" (UID: "f566a17f-43cf-481d-9732-2ebc9bc71391"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.338002 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.337959    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-ca" pod "734f8b87-0bbd-4961-8cbd-867fe23f2244" (UID: "734f8b87-0bbd-4961-8cbd-867fe23f2244"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.338965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.338687    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-config" pod "6f84781d-58a8-422d-a90f-60692f2ad4f6" (UID: "6f84781d-58a8-422d-a90f-60692f2ad4f6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.338965 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.338930    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-prometheus-tls-ca" pod "6f84781d-58a8-422d-a90f-60692f2ad4f6" (UID: "6f84781d-58a8-422d-a90f-60692f2ad4f6"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.339387 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.339351    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-client-ca" pod "6f84781d-58a8-422d-a90f-60692f2ad4f6" (UID: "6f84781d-58a8-422d-a90f-60692f2ad4f6"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:52.342439 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.342378    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert" pod "734f8b87-0bbd-4961-8cbd-867fe23f2244" (UID: "734f8b87-0bbd-4961-8cbd-867fe23f2244"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.342439 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.342401    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-rbac-proxy-config" pod "734f8b87-0bbd-4961-8cbd-867fe23f2244" (UID: "734f8b87-0bbd-4961-8cbd-867fe23f2244"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.342644 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.342490    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f566a17f-43cf-481d-9732-2ebc9bc71391-tls" pod "f566a17f-43cf-481d-9732-2ebc9bc71391" (UID: "f566a17f-43cf-481d-9732-2ebc9bc71391"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.342715 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.342677    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f566a17f-43cf-481d-9732-2ebc9bc71391-kube-api-access-lrh9x" pod "f566a17f-43cf-481d-9732-2ebc9bc71391" (UID: "f566a17f-43cf-481d-9732-2ebc9bc71391"). InnerVolumeSpecName "kube-api-access-lrh9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:52.343024 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.342997    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-api-access-d8bp8" pod "734f8b87-0bbd-4961-8cbd-867fe23f2244" (UID: "734f8b87-0bbd-4961-8cbd-867fe23f2244"). InnerVolumeSpecName "kube-api-access-d8bp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:52.344837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.344810    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-tls" pod "734f8b87-0bbd-4961-8cbd-867fe23f2244" (UID: "734f8b87-0bbd-4961-8cbd-867fe23f2244"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.345613 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.345590    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f84781d-58a8-422d-a90f-60692f2ad4f6-tls" pod "6f84781d-58a8-422d-a90f-60692f2ad4f6" (UID: "6f84781d-58a8-422d-a90f-60692f2ad4f6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:52.345950 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.345929    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f84781d-58a8-422d-a90f-60692f2ad4f6-kube-api-access-tvtmb" pod "6f84781d-58a8-422d-a90f-60692f2ad4f6" (UID: "6f84781d-58a8-422d-a90f-60692f2ad4f6"). InnerVolumeSpecName "kube-api-access-tvtmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:52.438661 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438630    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438661 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438657    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f566a17f-43cf-481d-9732-2ebc9bc71391-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438661 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438666    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f566a17f-43cf-481d-9732-2ebc9bc71391-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438674    4334 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-rbac-proxy-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438685    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtmb\" (UniqueName: \"kubernetes.io/projected/6f84781d-58a8-422d-a90f-60692f2ad4f6-kube-api-access-tvtmb\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438693    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438701    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrh9x\" (UniqueName: \"kubernetes.io/projected/f566a17f-43cf-481d-9732-2ebc9bc71391-kube-api-access-lrh9x\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438710    4334 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-client-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438718    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438726    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/734f8b87-0bbd-4961-8cbd-867fe23f2244-prometheus-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438734    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bp8\" (UniqueName: \"kubernetes.io/projected/734f8b87-0bbd-4961-8cbd-867fe23f2244-kube-api-access-d8bp8\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438743    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6f84781d-58a8-422d-a90f-60692f2ad4f6-prometheus-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.438999 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.438752    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6f84781d-58a8-422d-a90f-60692f2ad4f6-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:52.450607 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.450579    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:52.450897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.450856    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="alertmanager" containerID="cri-o://2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11" gracePeriod=120
Oct 30 04:16:52.450993 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.450881    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="config-reloader" containerID="cri-o://e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b" gracePeriod=120
Oct 30 04:16:52.464094 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.464066    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:52.465134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.464386    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="prometheus" containerID="cri-o://dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3" gracePeriod=600
Oct 30 04:16:52.465134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.464566    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="thanos-sidecar" containerID="cri-o://2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54" gracePeriod=600
Oct 30 04:16:52.465134 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.464657    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="config-reloader" containerID="cri-o://f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3" gracePeriod=600
Oct 30 04:16:52.470564 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.470539    4334 generic.go:356] "Generic (PLEG): container finished" podID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerID="bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8" exitCode=0
Oct 30 04:16:52.470564 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.470563    4334 generic.go:356] "Generic (PLEG): container finished" podID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerID="439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9" exitCode=0
Oct 30 04:16:52.470764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.470604    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"
Oct 30 04:16:52.470764 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.470711    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" event={"ID":"f566a17f-43cf-481d-9732-2ebc9bc71391","Type":"ContainerDied","Data":"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8"}
Oct 30 04:16:52.470879 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.470768    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" event={"ID":"f566a17f-43cf-481d-9732-2ebc9bc71391","Type":"ContainerDied","Data":"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9"}
Oct 30 04:16:52.470879 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.470802    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6" event={"ID":"f566a17f-43cf-481d-9732-2ebc9bc71391","Type":"ContainerDied","Data":"16c7291234650194f6486e30ec6b3d13008b9119191fab7b12e60053f1f0795c"}
Oct 30 04:16:52.470879 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.470838    4334 scope.go:122] "RemoveContainer" containerID="bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8"
Oct 30 04:16:52.473299 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.473276    4334 generic.go:356] "Generic (PLEG): container finished" podID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerID="0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c" exitCode=0
Oct 30 04:16:52.473603 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.473586    4334 generic.go:356] "Generic (PLEG): container finished" podID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerID="4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03" exitCode=0
Oct 30 04:16:52.473821 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.473544    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"
Oct 30 04:16:52.473975 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.473506    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" event={"ID":"6f84781d-58a8-422d-a90f-60692f2ad4f6","Type":"ContainerDied","Data":"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c"}
Oct 30 04:16:52.473975 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.473947    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" event={"ID":"6f84781d-58a8-422d-a90f-60692f2ad4f6","Type":"ContainerDied","Data":"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03"}
Oct 30 04:16:52.474150 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.473979    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2" event={"ID":"6f84781d-58a8-422d-a90f-60692f2ad4f6","Type":"ContainerDied","Data":"fbc63ac3bcd04278cc686d651a9c86724524656f5df40a2faac58e9d9676cf8f"}
Oct 30 04:16:52.478361 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.478337    4334 generic.go:356] "Generic (PLEG): container finished" podID="734f8b87-0bbd-4961-8cbd-867fe23f2244" containerID="c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb" exitCode=0
Oct 30 04:16:52.478499 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.478390    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk" event={"ID":"734f8b87-0bbd-4961-8cbd-867fe23f2244","Type":"ContainerDied","Data":"c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb"}
Oct 30 04:16:52.478499 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.478432    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk" event={"ID":"734f8b87-0bbd-4961-8cbd-867fe23f2244","Type":"ContainerDied","Data":"715e9a3d17e2f70ad1d3dca24f6c817c2c8a9f5f02cd1d94d9694f8a2f0fa7c0"}
Oct 30 04:16:52.478644 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.478516    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"
Oct 30 04:16:52.530549 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.530496    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"]
Oct 30 04:16:52.534487 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.534463    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-v6lvk"]
Oct 30 04:16:52.562511 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.562491    4334 scope.go:122] "RemoveContainer" containerID="439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9"
Oct 30 04:16:52.584963 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.584944    4334 scope.go:122] "RemoveContainer" containerID="bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8"
Oct 30 04:16:52.585251 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:52.585231    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8\": container with ID starting with bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8 not found: ID does not exist" containerID="bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8"
Oct 30 04:16:52.585310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.585260    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8"} err="failed to get container status \"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8\": rpc error: code = NotFound desc = could not find container \"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8\": container with ID starting with bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8 not found: ID does not exist"
Oct 30 04:16:52.585310 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.585282    4334 scope.go:122] "RemoveContainer" containerID="439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9"
Oct 30 04:16:52.585561 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:52.585544    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9\": container with ID starting with 439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9 not found: ID does not exist" containerID="439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9"
Oct 30 04:16:52.585621 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.585566    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9"} err="failed to get container status \"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9\": rpc error: code = NotFound desc = could not find container \"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9\": container with ID starting with 439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9 not found: ID does not exist"
Oct 30 04:16:52.585621 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.585580    4334 scope.go:122] "RemoveContainer" containerID="bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8"
Oct 30 04:16:52.585806 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.585790    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8"} err="failed to get container status \"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8\": rpc error: code = NotFound desc = could not find container \"bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8\": container with ID starting with bd6d593ea9ea79840c0aead15413bb5f4c133fee944191fedb6978907a5d72b8 not found: ID does not exist"
Oct 30 04:16:52.585859 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.585807    4334 scope.go:122] "RemoveContainer" containerID="439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9"
Oct 30 04:16:52.586011 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.585996    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9"} err="failed to get container status \"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9\": rpc error: code = NotFound desc = could not find container \"439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9\": container with ID starting with 439ee4eb765407e663a297b9e6ac24ff3a2dfadf6aed2580fffd3a2b8cf545f9 not found: ID does not exist"
Oct 30 04:16:52.586066 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.586011    4334 scope.go:122] "RemoveContainer" containerID="0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c"
Oct 30 04:16:52.593695 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.593680    4334 scope.go:122] "RemoveContainer" containerID="4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03"
Oct 30 04:16:52.601042 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601025    4334 scope.go:122] "RemoveContainer" containerID="0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c"
Oct 30 04:16:52.601292 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:52.601277    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c\": container with ID starting with 0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c not found: ID does not exist" containerID="0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c"
Oct 30 04:16:52.601342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601299    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c"} err="failed to get container status \"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c\": rpc error: code = NotFound desc = could not find container \"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c\": container with ID starting with 0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c not found: ID does not exist"
Oct 30 04:16:52.601342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601312    4334 scope.go:122] "RemoveContainer" containerID="4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03"
Oct 30 04:16:52.601540 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:52.601524    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03\": container with ID starting with 4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03 not found: ID does not exist" containerID="4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03"
Oct 30 04:16:52.601588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601547    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03"} err="failed to get container status \"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03\": rpc error: code = NotFound desc = could not find container \"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03\": container with ID starting with 4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03 not found: ID does not exist"
Oct 30 04:16:52.601588 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601559    4334 scope.go:122] "RemoveContainer" containerID="0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c"
Oct 30 04:16:52.601800 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601783    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c"} err="failed to get container status \"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c\": rpc error: code = NotFound desc = could not find container \"0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c\": container with ID starting with 0ff5b6761573c43cf33c3b504b2af0de32335d26ad0fc16634956a760efb798c not found: ID does not exist"
Oct 30 04:16:52.601800 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601799    4334 scope.go:122] "RemoveContainer" containerID="4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03"
Oct 30 04:16:52.601998 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601982    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03"} err="failed to get container status \"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03\": rpc error: code = NotFound desc = could not find container \"4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03\": container with ID starting with 4c308850a9f04c23b1a888f79fe9190e0954b50f21cfe1eea8f7f16c84799a03 not found: ID does not exist"
Oct 30 04:16:52.602046 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.601998    4334 scope.go:122] "RemoveContainer" containerID="c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb"
Oct 30 04:16:52.608769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.608744    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"]
Oct 30 04:16:52.609586 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.609572    4334 scope.go:122] "RemoveContainer" containerID="c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb"
Oct 30 04:16:52.614408 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:52.614381    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb\": container with ID starting with c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb not found: ID does not exist" containerID="c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb"
Oct 30 04:16:52.614557 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.614414    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb"} err="failed to get container status \"c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb\": rpc error: code = NotFound desc = could not find container \"c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb\": container with ID starting with c2a232ceba087bd195c0ca30fc5ff53c51d14fca41ae3b2f630c50e421e261bb not found: ID does not exist"
Oct 30 04:16:52.616381 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.616360    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-drwv2"]
Oct 30 04:16:52.647139 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.647116    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"]
Oct 30 04:16:52.650272 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.650251    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-jbrf6"]
Oct 30 04:16:52.960379 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:52.960358    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:53.047020 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.046992    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-data-science-collector-ta-client-cert\") pod \"49777d61-2a10-4d15-a980-d5ce6618b44d\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") "
Oct 30 04:16:53.047020 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.047022    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-monitoring-tls-certs\") pod \"49777d61-2a10-4d15-a980-d5ce6618b44d\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") "
Oct 30 04:16:53.047266 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.047058    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49777d61-2a10-4d15-a980-d5ce6618b44d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/49777d61-2a10-4d15-a980-d5ce6618b44d-otc-internal\") pod \"49777d61-2a10-4d15-a980-d5ce6618b44d\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") "
Oct 30 04:16:53.047266 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.047123    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-otlp-tls-certs\") pod \"49777d61-2a10-4d15-a980-d5ce6618b44d\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") "
Oct 30 04:16:53.047266 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.047147    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-tls-certs\") pod \"49777d61-2a10-4d15-a980-d5ce6618b44d\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") "
Oct 30 04:16:53.047266 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.047171    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49777d61-2a10-4d15-a980-d5ce6618b44d-kube-api-access-9448x\" (UniqueName: \"kubernetes.io/projected/49777d61-2a10-4d15-a980-d5ce6618b44d-kube-api-access-9448x\") pod \"49777d61-2a10-4d15-a980-d5ce6618b44d\" (UID: \"49777d61-2a10-4d15-a980-d5ce6618b44d\") "
Oct 30 04:16:53.047597 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.047564    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49777d61-2a10-4d15-a980-d5ce6618b44d-otc-internal" pod "49777d61-2a10-4d15-a980-d5ce6618b44d" (UID: "49777d61-2a10-4d15-a980-d5ce6618b44d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:53.051895 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.051868    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-tls-certs" pod "49777d61-2a10-4d15-a980-d5ce6618b44d" (UID: "49777d61-2a10-4d15-a980-d5ce6618b44d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.052007 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.051973    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-data-science-collector-ta-client-cert" pod "49777d61-2a10-4d15-a980-d5ce6618b44d" (UID: "49777d61-2a10-4d15-a980-d5ce6618b44d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.052092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.052001    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-otlp-tls-certs" pod "49777d61-2a10-4d15-a980-d5ce6618b44d" (UID: "49777d61-2a10-4d15-a980-d5ce6618b44d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.052152 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.052088    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49777d61-2a10-4d15-a980-d5ce6618b44d-kube-api-access-9448x" pod "49777d61-2a10-4d15-a980-d5ce6618b44d" (UID: "49777d61-2a10-4d15-a980-d5ce6618b44d"). InnerVolumeSpecName "kube-api-access-9448x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:53.052188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.052168    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-monitoring-tls-certs" pod "49777d61-2a10-4d15-a980-d5ce6618b44d" (UID: "49777d61-2a10-4d15-a980-d5ce6618b44d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.148486 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.147946    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.148486 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.147981    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.148486 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.147998    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9448x\" (UniqueName: \"kubernetes.io/projected/49777d61-2a10-4d15-a980-d5ce6618b44d-kube-api-access-9448x\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.148486 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.148015    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.148486 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.148029    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49777d61-2a10-4d15-a980-d5ce6618b44d-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.148486 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.148042    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/49777d61-2a10-4d15-a980-d5ce6618b44d-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.166658 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.162432    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" path="/var/lib/kubelet/pods/6f84781d-58a8-422d-a90f-60692f2ad4f6/volumes"
Oct 30 04:16:53.166658 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.163067    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734f8b87-0bbd-4961-8cbd-867fe23f2244" path="/var/lib/kubelet/pods/734f8b87-0bbd-4961-8cbd-867fe23f2244/volumes"
Oct 30 04:16:53.166658 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.163557    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" path="/var/lib/kubelet/pods/f566a17f-43cf-481d-9732-2ebc9bc71391/volumes"
Oct 30 04:16:53.210221 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.210194    4334 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25bc8f4_d802_4013_8912_57b3149d79c4.slice/crio-conmon-2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 04:16:53.210474 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.210223    4334 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25bc8f4_d802_4013_8912_57b3149d79c4.slice/crio-conmon-2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 04:16:53.277953 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.277930    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:53.349699 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.349623    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.349699 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.349680    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.349919 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.349726    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-config\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.349919 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.349760    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-tls-assets\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-tls-assets\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.349919 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.349860    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.349919 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.349892    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.350151 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.349939    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-secret-prometheus-web-tls-ca\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.350151 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.350000    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.350151 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.350053    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-kube-api-access-7px89\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-kube-api-access-7px89\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.350151 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.350091    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-thanos-prometheus-http-client-file\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.350151 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.350123    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.350423 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.350158    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5430e9-3440-4a11-b4bd-68583ca2864d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f5430e9-3440-4a11-b4bd-68583ca2864d-config-out\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.350423 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.350190    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"1f5430e9-3440-4a11-b4bd-68583ca2864d\" (UID: \"1f5430e9-3440-4a11-b4bd-68583ca2864d\") "
Oct 30 04:16:53.350730 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.350702    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-2" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:53.350730 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.350712    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-0" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:53.354021 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.353791    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-kube-api-access-7px89" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "kube-api-access-7px89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:53.354947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.354710    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-tls-assets" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:53.356817 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.356788    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-thanos-prometheus-http-client-file" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.357122 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.357101    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:16:53.357843 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.357820    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-config" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.360772 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.360694    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.361030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.361003    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-1" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:53.361416 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.361391    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.361719 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.361697    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-secret-prometheus-web-tls-ca" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.362399 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.362374    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5430e9-3440-4a11-b4bd-68583ca2864d-config-out" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.370897 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.370873    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config" pod "1f5430e9-3440-4a11-b4bd-68583ca2864d" (UID: "1f5430e9-3440-4a11-b4bd-68583ca2864d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.402214 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.402190    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:53.453849 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.452540    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-kube-api-access-kg5xh\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-kube-api-access-kg5xh\") pod \"c25bc8f4-d802-4013-8912-57b3149d79c4\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") "
Oct 30 04:16:53.453849 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.452640    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-config-out\") pod \"c25bc8f4-d802-4013-8912-57b3149d79c4\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") "
Oct 30 04:16:53.453849 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.452685    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-alertmanager-data-science-monitoringstack-db\") pod \"c25bc8f4-d802-4013-8912-57b3149d79c4\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") "
Oct 30 04:16:53.453849 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.452718    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-web-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-web-config\") pod \"c25bc8f4-d802-4013-8912-57b3149d79c4\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") "
Oct 30 04:16:53.456119 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456096    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-config-volume\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-config-volume\") pod \"c25bc8f4-d802-4013-8912-57b3149d79c4\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") "
Oct 30 04:16:53.456250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456129    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-tls-assets\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-tls-assets\") pod \"c25bc8f4-d802-4013-8912-57b3149d79c4\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") "
Oct 30 04:16:53.456250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456167    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-alertmanager-data-science-monitoringstack-db" pod "c25bc8f4-d802-4013-8912-57b3149d79c4" (UID: "c25bc8f4-d802-4013-8912-57b3149d79c4"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.456250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456208    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-cluster-tls-config\") pod \"c25bc8f4-d802-4013-8912-57b3149d79c4\" (UID: \"c25bc8f4-d802-4013-8912-57b3149d79c4\") "
Oct 30 04:16:53.456960 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456836    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.456960 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456888    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f5430e9-3440-4a11-b4bd-68583ca2864d-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.456960 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456906    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.456960 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456923    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.456947    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458266    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458285    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458325    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458340    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458356    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458371    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f5430e9-3440-4a11-b4bd-68583ca2864d-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458387    4334 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458403    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7px89\" (UniqueName: \"kubernetes.io/projected/1f5430e9-3440-4a11-b4bd-68583ca2864d-kube-api-access-7px89\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.458474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458417    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f5430e9-3440-4a11-b4bd-68583ca2864d-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.459058 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.458598    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-config-out" pod "c25bc8f4-d802-4013-8912-57b3149d79c4" (UID: "c25bc8f4-d802-4013-8912-57b3149d79c4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.459753 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.459718    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-config-volume" pod "c25bc8f4-d802-4013-8912-57b3149d79c4" (UID: "c25bc8f4-d802-4013-8912-57b3149d79c4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.459848 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.459826    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-kube-api-access-kg5xh" pod "c25bc8f4-d802-4013-8912-57b3149d79c4" (UID: "c25bc8f4-d802-4013-8912-57b3149d79c4"). InnerVolumeSpecName "kube-api-access-kg5xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:53.460710 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.460685    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-tls-assets" pod "c25bc8f4-d802-4013-8912-57b3149d79c4" (UID: "c25bc8f4-d802-4013-8912-57b3149d79c4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:53.463953 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.463932    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-cluster-tls-config" pod "c25bc8f4-d802-4013-8912-57b3149d79c4" (UID: "c25bc8f4-d802-4013-8912-57b3149d79c4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.469427 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.469408    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-web-config" pod "c25bc8f4-d802-4013-8912-57b3149d79c4" (UID: "c25bc8f4-d802-4013-8912-57b3149d79c4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.470587 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.470572    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:16:53.484716 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.484695    4334 generic.go:356] "Generic (PLEG): container finished" podID="49777d61-2a10-4d15-a980-d5ce6618b44d" containerID="5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b" exitCode=0
Oct 30 04:16:53.484812 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.484751    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:53.484812 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.484770    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"49777d61-2a10-4d15-a980-d5ce6618b44d","Type":"ContainerDied","Data":"5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b"}
Oct 30 04:16:53.484912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.484850    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"49777d61-2a10-4d15-a980-d5ce6618b44d","Type":"ContainerDied","Data":"7544579b6cf5b54938c52252f95dc590f916988aaf2a6bd335849ea6b7b4f2c7"}
Oct 30 04:16:53.484912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.484884    4334 scope.go:122] "RemoveContainer" containerID="5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b"
Oct 30 04:16:53.488276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.488251    4334 generic.go:356] "Generic (PLEG): container finished" podID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerID="2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54" exitCode=0
Oct 30 04:16:53.488276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.488275    4334 generic.go:356] "Generic (PLEG): container finished" podID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerID="f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3" exitCode=0
Oct 30 04:16:53.488434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.488286    4334 generic.go:356] "Generic (PLEG): container finished" podID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerID="dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3" exitCode=0
Oct 30 04:16:53.488434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.488324    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerDied","Data":"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"}
Oct 30 04:16:53.488434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.488339    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:16:53.488434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.488373    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerDied","Data":"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"}
Oct 30 04:16:53.488434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.488405    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerDied","Data":"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"}
Oct 30 04:16:53.488434 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.488434    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"1f5430e9-3440-4a11-b4bd-68583ca2864d","Type":"ContainerDied","Data":"80f5014ecdcbf3d10c56fa5b88739361e37641abf55fe05b167d2964356d614d"}
Oct 30 04:16:53.491279 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.491255    4334 generic.go:356] "Generic (PLEG): container finished" podID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerID="e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b" exitCode=0
Oct 30 04:16:53.491381 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.491284    4334 generic.go:356] "Generic (PLEG): container finished" podID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerID="2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11" exitCode=0
Oct 30 04:16:53.491381 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.491320    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:16:53.491381 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.491331    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c25bc8f4-d802-4013-8912-57b3149d79c4","Type":"ContainerDied","Data":"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b"}
Oct 30 04:16:53.491381 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.491375    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c25bc8f4-d802-4013-8912-57b3149d79c4","Type":"ContainerDied","Data":"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11"}
Oct 30 04:16:53.491653 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.491397    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c25bc8f4-d802-4013-8912-57b3149d79c4","Type":"ContainerDied","Data":"a4a72e787e8b89693cbc105fbedaed17d6cdf147cd0e489be966c9358d2eb8cf"}
Oct 30 04:16:53.494412 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.494392    4334 scope.go:122] "RemoveContainer" containerID="5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b"
Oct 30 04:16:53.494710 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.494690    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b\": container with ID starting with 5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b not found: ID does not exist" containerID="5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b"
Oct 30 04:16:53.494842 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.494718    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b"} err="failed to get container status \"5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b\": rpc error: code = NotFound desc = could not find container \"5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b\": container with ID starting with 5993b856daf686507bbe4f24cdb3df38eb24564a3d71f45995984c6bb5b5184b not found: ID does not exist"
Oct 30 04:16:53.494842 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.494738    4334 scope.go:122] "RemoveContainer" containerID="2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"
Oct 30 04:16:53.502672 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.502652    4334 scope.go:122] "RemoveContainer" containerID="f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"
Oct 30 04:16:53.510474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.510459    4334 scope.go:122] "RemoveContainer" containerID="dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"
Oct 30 04:16:53.519566 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.519548    4334 scope.go:122] "RemoveContainer" containerID="e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"
Oct 30 04:16:53.527168 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.527142    4334 scope.go:122] "RemoveContainer" containerID="2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"
Oct 30 04:16:53.527423 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.527403    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54\": container with ID starting with 2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54 not found: ID does not exist" containerID="2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"
Oct 30 04:16:53.527513 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.527428    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"} err="failed to get container status \"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54\": rpc error: code = NotFound desc = could not find container \"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54\": container with ID starting with 2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54 not found: ID does not exist"
Oct 30 04:16:53.527513 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.527480    4334 scope.go:122] "RemoveContainer" containerID="f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"
Oct 30 04:16:53.527751 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.527732    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3\": container with ID starting with f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3 not found: ID does not exist" containerID="f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"
Oct 30 04:16:53.527805 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.527756    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"} err="failed to get container status \"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3\": rpc error: code = NotFound desc = could not find container \"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3\": container with ID starting with f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3 not found: ID does not exist"
Oct 30 04:16:53.527805 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.527771    4334 scope.go:122] "RemoveContainer" containerID="dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"
Oct 30 04:16:53.528046 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.528028    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3\": container with ID starting with dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3 not found: ID does not exist" containerID="dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"
Oct 30 04:16:53.528089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.528052    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"} err="failed to get container status \"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3\": rpc error: code = NotFound desc = could not find container \"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3\": container with ID starting with dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3 not found: ID does not exist"
Oct 30 04:16:53.528089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.528069    4334 scope.go:122] "RemoveContainer" containerID="e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"
Oct 30 04:16:53.528314 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.528297    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390\": container with ID starting with e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390 not found: ID does not exist" containerID="e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"
Oct 30 04:16:53.528358 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.528320    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"} err="failed to get container status \"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390\": rpc error: code = NotFound desc = could not find container \"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390\": container with ID starting with e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390 not found: ID does not exist"
Oct 30 04:16:53.528358 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.528342    4334 scope.go:122] "RemoveContainer" containerID="2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"
Oct 30 04:16:53.528722 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.528705    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"} err="failed to get container status \"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54\": rpc error: code = NotFound desc = could not find container \"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54\": container with ID starting with 2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54 not found: ID does not exist"
Oct 30 04:16:53.528771 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.528722    4334 scope.go:122] "RemoveContainer" containerID="f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"
Oct 30 04:16:53.528976 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.528960    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"} err="failed to get container status \"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3\": rpc error: code = NotFound desc = could not find container \"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3\": container with ID starting with f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3 not found: ID does not exist"
Oct 30 04:16:53.528976 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.528975    4334 scope.go:122] "RemoveContainer" containerID="dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"
Oct 30 04:16:53.529197 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.529181    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"} err="failed to get container status \"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3\": rpc error: code = NotFound desc = could not find container \"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3\": container with ID starting with dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3 not found: ID does not exist"
Oct 30 04:16:53.529197 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.529197    4334 scope.go:122] "RemoveContainer" containerID="e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"
Oct 30 04:16:53.529422 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.529408    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"} err="failed to get container status \"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390\": rpc error: code = NotFound desc = could not find container \"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390\": container with ID starting with e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390 not found: ID does not exist"
Oct 30 04:16:53.529499 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.529421    4334 scope.go:122] "RemoveContainer" containerID="2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"
Oct 30 04:16:53.529663 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.529648    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54"} err="failed to get container status \"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54\": rpc error: code = NotFound desc = could not find container \"2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54\": container with ID starting with 2779333e8eadda12d453a40f627ca53db5c6213986450f90aca449246b11cf54 not found: ID does not exist"
Oct 30 04:16:53.529703 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.529663    4334 scope.go:122] "RemoveContainer" containerID="f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"
Oct 30 04:16:53.529855 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.529837    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3"} err="failed to get container status \"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3\": rpc error: code = NotFound desc = could not find container \"f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3\": container with ID starting with f25185edce687dd5e2c52b333f2643ef526aa13ad7b802a520083f0e046b08b3 not found: ID does not exist"
Oct 30 04:16:53.529894 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.529855    4334 scope.go:122] "RemoveContainer" containerID="dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"
Oct 30 04:16:53.530059 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.530043    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3"} err="failed to get container status \"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3\": rpc error: code = NotFound desc = could not find container \"dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3\": container with ID starting with dc83d5e83f98e835f0d9a04e06d1a7bd65ff348736dc6cb8cdeac08f94f279c3 not found: ID does not exist"
Oct 30 04:16:53.530101 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.530058    4334 scope.go:122] "RemoveContainer" containerID="e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"
Oct 30 04:16:53.530249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.530232    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390"} err="failed to get container status \"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390\": rpc error: code = NotFound desc = could not find container \"e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390\": container with ID starting with e354666e1373113f53dc309d210aabdae9106668c418dd500f5f757444b71390 not found: ID does not exist"
Oct 30 04:16:53.530292 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.530249    4334 scope.go:122] "RemoveContainer" containerID="e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b"
Oct 30 04:16:53.537170 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.537154    4334 scope.go:122] "RemoveContainer" containerID="2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11"
Oct 30 04:16:53.544859 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.544839    4334 scope.go:122] "RemoveContainer" containerID="61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6"
Oct 30 04:16:53.551739 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.551721    4334 scope.go:122] "RemoveContainer" containerID="e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b"
Oct 30 04:16:53.551995 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.551978    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b\": container with ID starting with e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b not found: ID does not exist" containerID="e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b"
Oct 30 04:16:53.552066 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.552004    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b"} err="failed to get container status \"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b\": rpc error: code = NotFound desc = could not find container \"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b\": container with ID starting with e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b not found: ID does not exist"
Oct 30 04:16:53.552066 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.552024    4334 scope.go:122] "RemoveContainer" containerID="2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11"
Oct 30 04:16:53.552268 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.552252    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11\": container with ID starting with 2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11 not found: ID does not exist" containerID="2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11"
Oct 30 04:16:53.552305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.552273    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11"} err="failed to get container status \"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11\": rpc error: code = NotFound desc = could not find container \"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11\": container with ID starting with 2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11 not found: ID does not exist"
Oct 30 04:16:53.552305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.552288    4334 scope.go:122] "RemoveContainer" containerID="61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6"
Oct 30 04:16:53.552536 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:16:53.552518    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6\": container with ID starting with 61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6 not found: ID does not exist" containerID="61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6"
Oct 30 04:16:53.552615 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.552543    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6"} err="failed to get container status \"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6\": rpc error: code = NotFound desc = could not find container \"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6\": container with ID starting with 61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6 not found: ID does not exist"
Oct 30 04:16:53.552615 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.552560    4334 scope.go:122] "RemoveContainer" containerID="e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b"
Oct 30 04:16:53.552818 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.552800    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b"} err="failed to get container status \"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b\": rpc error: code = NotFound desc = could not find container \"e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b\": container with ID starting with e1ebfbc6b21b20e4cf8794e9f8c4160f885cf71620ce5a7197a0a4cefb89e74b not found: ID does not exist"
Oct 30 04:16:53.552865 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.552819    4334 scope.go:122] "RemoveContainer" containerID="2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11"
Oct 30 04:16:53.553024 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.553005    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11"} err="failed to get container status \"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11\": rpc error: code = NotFound desc = could not find container \"2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11\": container with ID starting with 2fea67775d20755731a366709d11a3f2ea736e62f8e8ed2ec34135870e786a11 not found: ID does not exist"
Oct 30 04:16:53.553089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.553024    4334 scope.go:122] "RemoveContainer" containerID="61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6"
Oct 30 04:16:53.553249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.553231    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6"} err="failed to get container status \"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6\": rpc error: code = NotFound desc = could not find container \"61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6\": container with ID starting with 61a1adddc50e4573a7e26bae0a23ad55f971e87084578bcce65f05cf2627e0e6 not found: ID does not exist"
Oct 30 04:16:53.559611 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.559594    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.559662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.559613    4334 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-config-volume\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.559662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.559622    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.559662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.559630    4334 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c25bc8f4-d802-4013-8912-57b3149d79c4-cluster-tls-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.559662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.559638    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5xh\" (UniqueName: \"kubernetes.io/projected/c25bc8f4-d802-4013-8912-57b3149d79c4-kube-api-access-kg5xh\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.559662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.559648    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\""
Oct 30 04:16:53.559662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.559657    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c25bc8f4-d802-4013-8912-57b3149d79c4-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.564002 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.563952    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:53.567741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.567720    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:53.601040 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.600984    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:53.604317 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.604299    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:16:53.636958 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.636937    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:53.640420 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:53.640403    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:16:55.158016 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:55.157987    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" path="/var/lib/kubelet/pods/1f5430e9-3440-4a11-b4bd-68583ca2864d/volumes"
Oct 30 04:16:55.158499 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:55.158486    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49777d61-2a10-4d15-a980-d5ce6618b44d" path="/var/lib/kubelet/pods/49777d61-2a10-4d15-a980-d5ce6618b44d/volumes"
Oct 30 04:16:55.159058 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:16:55.159044    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" path="/var/lib/kubelet/pods/c25bc8f4-d802-4013-8912-57b3149d79c4/volumes"
Oct 30 04:17:00.982013 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.981977    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"]
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982306    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerName="kube-rbac-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982326    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerName="kube-rbac-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982344    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerName="kube-rbac-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982349    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerName="kube-rbac-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982364    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="thanos-sidecar"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982369    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="thanos-sidecar"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982377    4334 cpu_manager.go:414] "containerMap: removing container" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerName="prom-label-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982382    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerName="prom-label-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982388    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="prometheus"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982394    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="prometheus"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982407    4334 cpu_manager.go:414] "containerMap: removing container" podUID="734f8b87-0bbd-4961-8cbd-867fe23f2244" containerName="kube-rbac-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982414    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734f8b87-0bbd-4961-8cbd-867fe23f2244" containerName="kube-rbac-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982419    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="init-config-reloader"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982424    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="init-config-reloader"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982430    4334 cpu_manager.go:414] "containerMap: removing container" podUID="49777d61-2a10-4d15-a980-d5ce6618b44d" containerName="otc-container"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982434    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49777d61-2a10-4d15-a980-d5ce6618b44d" containerName="otc-container"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982442    4334 cpu_manager.go:414] "containerMap: removing container" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerName="prom-label-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982460    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerName="prom-label-proxy"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982470    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="init-config-reloader"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982477    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="init-config-reloader"
Oct 30 04:17:00.982483 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982487    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="config-reloader"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982496    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="config-reloader"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982504    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="config-reloader"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982510    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="config-reloader"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982520    4334 cpu_manager.go:414] "containerMap: removing container" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="alertmanager"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982524    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="alertmanager"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982576    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="prometheus"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982585    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="734f8b87-0bbd-4961-8cbd-867fe23f2244" containerName="kube-rbac-proxy"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982591    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="49777d61-2a10-4d15-a980-d5ce6618b44d" containerName="otc-container"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982597    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerName="prom-label-proxy"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982602    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="config-reloader"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982607    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="config-reloader"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982612    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25bc8f4-d802-4013-8912-57b3149d79c4" containerName="alertmanager"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982617    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="f566a17f-43cf-481d-9732-2ebc9bc71391" containerName="kube-rbac-proxy"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982623    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerName="kube-rbac-proxy"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982629    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5430e9-3440-4a11-b4bd-68583ca2864d" containerName="thanos-sidecar"
Oct 30 04:17:00.983131 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.982634    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f84781d-58a8-422d-a90f-60692f2ad4f6" containerName="prom-label-proxy"
Oct 30 04:17:00.987099 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:00.987084    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.003342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.003319    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\""
Oct 30 04:17:01.003502 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.003370    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:17:01.003860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.003812    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-rfs2z\""
Oct 30 04:17:01.003860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.003816    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:17:01.003860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.003833    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:17:01.003860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.003856    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\""
Oct 30 04:17:01.004115 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.003976    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:17:01.004643 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.004609    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"]
Oct 30 04:17:01.009239 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.009219    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.009336 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.009256    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.009336 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.009285    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6twl\" (UniqueName: \"kubernetes.io/projected/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-api-access-m6twl\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.009412 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.009392    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.009500 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.009486    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.033954 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.033932    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"]
Oct 30 04:17:01.037123 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.037108    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.050727 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.050710    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-5bhpg\""
Oct 30 04:17:01.055658 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.055638    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"]
Oct 30 04:17:01.072333 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.072309    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"]
Oct 30 04:17:01.075624 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.075608    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.097544 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.097522    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"]
Oct 30 04:17:01.102503 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.102482    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 04:17:01.102503 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.102480    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 04:17:01.102654 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.102507    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 04:17:01.102654 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.102522    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 04:17:01.102654 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.102624    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 04:17:01.102654 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.102635    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-7sjcn\""
Oct 30 04:17:01.110115 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110092    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b55p6\" (UniqueName: \"kubernetes.io/projected/2631e41e-25c4-422d-937e-40858114256e-kube-api-access-b55p6\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.110204 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110124    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.110204 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110146    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.110281 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110202    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.110281 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110236    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.110342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110277    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ca047f5-45c1-4513-81b8-7e131eed1dae-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.110342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110301    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnm7\" (UniqueName: \"kubernetes.io/projected/1ca047f5-45c1-4513-81b8-7e131eed1dae-kube-api-access-8gnm7\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.110342 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110329    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.110515 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:01.110502    4334 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found
Oct 30 04:17:01.110558 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:01.110551    4334 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert podName:2fd968fa-c141-49ec-b1e6-1732fa2acca1 nodeName:}" failed. No retries permitted until 2026-10-30 04:17:01.610533992 +0000 UTC m=+5700.617124228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-4m452" (UID: "2fd968fa-c141-49ec-b1e6-1732fa2acca1") : secret "prometheus-operated-tls" not found
Oct 30 04:17:01.110601 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110566    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6twl\" (UniqueName: \"kubernetes.io/projected/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-api-access-m6twl\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.110601 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110587    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.110686 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110611    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2631e41e-25c4-422d-937e-40858114256e-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.110686 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110639    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1ca047f5-45c1-4513-81b8-7e131eed1dae-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.110795 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110704    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.110893 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.110873    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.112843 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.112823    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.112899 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.112872    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.132928 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.132908    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6twl\" (UniqueName: \"kubernetes.io/projected/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-api-access-m6twl\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.211011 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.210980    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ca047f5-45c1-4513-81b8-7e131eed1dae-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.211011 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.211011    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnm7\" (UniqueName: \"kubernetes.io/projected/1ca047f5-45c1-4513-81b8-7e131eed1dae-kube-api-access-8gnm7\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.211260 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.211044    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2631e41e-25c4-422d-937e-40858114256e-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.211260 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.211073    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1ca047f5-45c1-4513-81b8-7e131eed1dae-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.211260 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.211092    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.211260 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.211114    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b55p6\" (UniqueName: \"kubernetes.io/projected/2631e41e-25c4-422d-937e-40858114256e-kube-api-access-b55p6\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.211260 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.211138    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.211260 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.211226    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.212089 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.212065    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.224778 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.224753    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 04:17:01.224876 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.224866    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 04:17:01.224938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.224878    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 04:17:01.224938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.224884    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 04:17:01.224938 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.224928    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 04:17:01.230600 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.230581    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnm7\" (UniqueName: \"kubernetes.io/projected/1ca047f5-45c1-4513-81b8-7e131eed1dae-kube-api-access-8gnm7\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.230689 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.230654    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b55p6\" (UniqueName: \"kubernetes.io/projected/2631e41e-25c4-422d-937e-40858114256e-kube-api-access-b55p6\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.231926 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.231892    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.231926 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.231899    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.232115 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.231998    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ca047f5-45c1-4513-81b8-7e131eed1dae-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.233429 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.233408    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2631e41e-25c4-422d-937e-40858114256e-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-2zlbh\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.233637 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.233621    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1ca047f5-45c1-4513-81b8-7e131eed1dae-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-cclpz\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.366799 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.366772    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-5bhpg\""
Oct 30 04:17:01.371860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.371843    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:01.403704 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.403680    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-7sjcn\""
Oct 30 04:17:01.410347 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.410261    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:01.614832 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.614745    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.617312 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.617287    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-4m452\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:01.680337 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.680303    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"]
Oct 30 04:17:01.688217 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:17:01.688187    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2631e41e_25c4_422d_937e_40858114256e.slice/crio-780829245dd85ca9692bf871ce24e3c6ba0ce16be888739b631020c157472cfc WatchSource:0}: Error finding container 780829245dd85ca9692bf871ce24e3c6ba0ce16be888739b631020c157472cfc: Status 404 returned error can't find the container with id 780829245dd85ca9692bf871ce24e3c6ba0ce16be888739b631020c157472cfc
Oct 30 04:17:01.716414 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.716389    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"]
Oct 30 04:17:01.720033 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:17:01.720007    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca047f5_45c1_4513_81b8_7e131eed1dae.slice/crio-bcf64f443713901a218f77147e6d2ac3448383459a6e3f6edb26513406193040 WatchSource:0}: Error finding container bcf64f443713901a218f77147e6d2ac3448383459a6e3f6edb26513406193040: Status 404 returned error can't find the container with id bcf64f443713901a218f77147e6d2ac3448383459a6e3f6edb26513406193040
Oct 30 04:17:01.912912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.912886    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-rfs2z\""
Oct 30 04:17:01.920664 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:01.920645    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:02.213881 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.213855    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"]
Oct 30 04:17:02.219936 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:17:02.219896    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd968fa_c141_49ec_b1e6_1732fa2acca1.slice/crio-fafbff845809ba6609a015d3ef4228c0529126bce20d36f8ff554a205586665a WatchSource:0}: Error finding container fafbff845809ba6609a015d3ef4228c0529126bce20d36f8ff554a205586665a: Status 404 returned error can't find the container with id fafbff845809ba6609a015d3ef4228c0529126bce20d36f8ff554a205586665a
Oct 30 04:17:02.538231 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.538146    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452" event={"ID":"2fd968fa-c141-49ec-b1e6-1732fa2acca1","Type":"ContainerStarted","Data":"fafbff845809ba6609a015d3ef4228c0529126bce20d36f8ff554a205586665a"}
Oct 30 04:17:02.540386 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.540357    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" event={"ID":"2631e41e-25c4-422d-937e-40858114256e","Type":"ContainerStarted","Data":"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1"}
Oct 30 04:17:02.540587 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.540411    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" event={"ID":"2631e41e-25c4-422d-937e-40858114256e","Type":"ContainerStarted","Data":"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95"}
Oct 30 04:17:02.540587 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.540458    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" event={"ID":"2631e41e-25c4-422d-937e-40858114256e","Type":"ContainerStarted","Data":"780829245dd85ca9692bf871ce24e3c6ba0ce16be888739b631020c157472cfc"}
Oct 30 04:17:02.542680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.542658    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" event={"ID":"1ca047f5-45c1-4513-81b8-7e131eed1dae","Type":"ContainerStarted","Data":"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b"}
Oct 30 04:17:02.542808 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.542704    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" event={"ID":"1ca047f5-45c1-4513-81b8-7e131eed1dae","Type":"ContainerStarted","Data":"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47"}
Oct 30 04:17:02.542808 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.542725    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" event={"ID":"1ca047f5-45c1-4513-81b8-7e131eed1dae","Type":"ContainerStarted","Data":"bcf64f443713901a218f77147e6d2ac3448383459a6e3f6edb26513406193040"}
Oct 30 04:17:02.579905 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.579847    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" podStartSLOduration=1.579828604 podStartE2EDuration="1.579828604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:02.579401452 +0000 UTC m=+5701.585991708" watchObservedRunningTime="2026-10-30 04:17:02.579828604 +0000 UTC m=+5701.586418859"
Oct 30 04:17:02.592573 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.592545    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:02.597415 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.597389    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.618880 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.618852    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:02.619231 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.619214    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-w4g8j\""
Oct 30 04:17:02.619358 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.619294    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:17:02.619358 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.619292    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\""
Oct 30 04:17:02.619358 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.619326    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\""
Oct 30 04:17:02.619546 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.619372    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:17:02.619546 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.619430    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:17:02.621942 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.621917    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.622056 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.621978    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.622123 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.622105    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.622184 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.622133    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.622184 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.622158    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.622285 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.622246    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v799g\" (UniqueName: \"kubernetes.io/projected/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-kube-api-access-v799g\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.623028 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.622980    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" podStartSLOduration=1.622965191 podStartE2EDuration="1.622965191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:02.620836501 +0000 UTC m=+5701.627426767" watchObservedRunningTime="2026-10-30 04:17:02.622965191 +0000 UTC m=+5701.629555447"
Oct 30 04:17:02.722988 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.722952    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.722988 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.722990    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.723275 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.723009    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.723275 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.723033    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v799g\" (UniqueName: \"kubernetes.io/projected/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-kube-api-access-v799g\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.723275 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.723067    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.723275 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.723086    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.723841 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.723817    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.725499 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.725461    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.725634 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.725561    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.725879 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.725858    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.726098 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.725971    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.745736 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.745713    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v799g\" (UniqueName: \"kubernetes.io/projected/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-kube-api-access-v799g\") pod \"data-science-collector-collector-0\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:02.920262 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:02.920235    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:03.212649 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:03.212611    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:03.218892 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:17:03.218867    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1cf36a_6e4d_4aa4_ba73_5609b4db5db3.slice/crio-f8c1d85a7817c35e704edf877aa0bf4dd126b3f3400b9bd288c335ec177851a5 WatchSource:0}: Error finding container f8c1d85a7817c35e704edf877aa0bf4dd126b3f3400b9bd288c335ec177851a5: Status 404 returned error can't find the container with id f8c1d85a7817c35e704edf877aa0bf4dd126b3f3400b9bd288c335ec177851a5
Oct 30 04:17:03.220983 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:03.220951    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:17:03.221099 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:03.221017    4334 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 04:17:03.554155 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:03.553705    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3","Type":"ContainerStarted","Data":"16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1"}
Oct 30 04:17:03.554155 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:03.553771    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3","Type":"ContainerStarted","Data":"f8c1d85a7817c35e704edf877aa0bf4dd126b3f3400b9bd288c335ec177851a5"}
Oct 30 04:17:03.555317 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:03.555291    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452" event={"ID":"2fd968fa-c141-49ec-b1e6-1732fa2acca1","Type":"ContainerStarted","Data":"b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0"}
Oct 30 04:17:03.593294 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:03.593235    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.5932169250000001 podStartE2EDuration="1.593216925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:03.592558668 +0000 UTC m=+5702.599148921" watchObservedRunningTime="2026-10-30 04:17:03.593216925 +0000 UTC m=+5702.599807180"
Oct 30 04:17:03.633754 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:03.632330    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452" podStartSLOduration=3.632313021 podStartE2EDuration="3.632313021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:03.629783098 +0000 UTC m=+5702.636373352" watchObservedRunningTime="2026-10-30 04:17:03.632313021 +0000 UTC m=+5702.638903276"
Oct 30 04:17:04.559947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:04.559911    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:06.232429 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.232397    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:17:06.236240 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.236223    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.272817 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.272789    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:17:06.341070 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.341038    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:17:06.341070 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.341038    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:17:06.341070 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.341075    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-rfjld\""
Oct 30 04:17:06.341070 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.341083    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:17:06.341335 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.341081    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:17:06.352574 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.352548    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.352694 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.352600    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.352749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.352703    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.352749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.352728    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.352851 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.352773    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.352912 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.352880    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmgv\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-kube-api-access-8mmgv\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.352968 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.352921    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.402842 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.402815    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:17:06.406874 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.406847    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.441204 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.441120    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:17:06.441514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.441441    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:17:06.441514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.441515    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:17:06.441685 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.441519    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:17:06.441685 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.441557    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:17:06.441685 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.441676    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-dbprj\""
Oct 30 04:17:06.443516 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.443491    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:17:06.453276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.453253    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.453369 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.453289    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.453369 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.453307    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.453369 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.453337    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.453625 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.453378    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmgv\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-kube-api-access-8mmgv\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.453625 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.453408    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.453625 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.453491    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.453817 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.453799    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.456133 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.456106    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.456334 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.456305    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.456438 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.456313    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.456438 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.456379    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.456438 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.456389    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.474121 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.474091    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmgv\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-kube-api-access-8mmgv\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.479297 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.479276    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:17:06.479391 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.479280    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:17:06.479504 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.479488    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:17:06.554231 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554172    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554231 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554203    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554231 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554226    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554247    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554325    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554368    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554402 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554387    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554565 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554416    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554565 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554495    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554565 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554556    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554579    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ck9\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-kube-api-access-c2ck9\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.554662 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.554614    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.556132 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.556115    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:06.655981 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.655943    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.655984    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656025    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656116 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656067    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656118    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656144    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ck9\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-kube-api-access-c2ck9\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656194    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656258    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656517 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656286    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656517 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656327    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656517 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656369    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656517 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656400    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.656860 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.656832    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.657163 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.657087    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659001 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.658741    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659301 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.659274    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659378 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.659315    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659378 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.659367    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659529 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.659381    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659652 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.659628    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659703 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.659662    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659703 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.659678    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.659983 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.659967    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.677028 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.677005    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ck9\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-kube-api-access-c2ck9\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:06.872372 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:06.872233    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:17:06.875381 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:17:06.875352    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c04cd0d_3f5a_4658_bd0d_8032d4195f55.slice/crio-2eee8a24da091de3c1c8b89d3fcdaad2de98662746e830251e18b955fd73690d WatchSource:0}: Error finding container 2eee8a24da091de3c1c8b89d3fcdaad2de98662746e830251e18b955fd73690d: Status 404 returned error can't find the container with id 2eee8a24da091de3c1c8b89d3fcdaad2de98662746e830251e18b955fd73690d
Oct 30 04:17:07.575667 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:07.575581    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3c04cd0d-3f5a-4658-bd0d-8032d4195f55","Type":"ContainerStarted","Data":"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8"}
Oct 30 04:17:07.575667 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:07.575641    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3c04cd0d-3f5a-4658-bd0d-8032d4195f55","Type":"ContainerStarted","Data":"2eee8a24da091de3c1c8b89d3fcdaad2de98662746e830251e18b955fd73690d"}
Oct 30 04:17:10.566634 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.566606    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:10.588641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.588610    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:10.625045 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.625016    4334 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:10.690173 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.690120    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:10.690385 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.690216    4334 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:10.703123 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.703096    4334 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:10.804665 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.804623    4334 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6b615f98501c6b5529f28e27a4cb6d663be03751269539a798523d989e4446fc/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:10.838071 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.837994    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:10.927474 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:10.927434    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:11.257098 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:11.257069    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:17:11.260055 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:17:11.260022    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ede8f0_7894_4b2c_9436_b0db653b8fcf.slice/crio-cfdbd39a0b87149ca25dc1d80b92b2b92c95a52ad3d9b3170abbf099c6822a38 WatchSource:0}: Error finding container cfdbd39a0b87149ca25dc1d80b92b2b92c95a52ad3d9b3170abbf099c6822a38: Status 404 returned error can't find the container with id cfdbd39a0b87149ca25dc1d80b92b2b92c95a52ad3d9b3170abbf099c6822a38
Oct 30 04:17:11.595677 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:11.595594    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerStarted","Data":"cfdbd39a0b87149ca25dc1d80b92b2b92c95a52ad3d9b3170abbf099c6822a38"}
Oct 30 04:17:11.597171 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:11.597147    4334 generic.go:356] "Generic (PLEG): container finished" podID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerID="b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8" exitCode=0
Oct 30 04:17:11.597263 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:11.597221    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3c04cd0d-3f5a-4658-bd0d-8032d4195f55","Type":"ContainerDied","Data":"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8"}
Oct 30 04:17:12.604160 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:12.604080    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerStarted","Data":"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"}
Oct 30 04:17:12.606323 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:12.606302    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3c04cd0d-3f5a-4658-bd0d-8032d4195f55","Type":"ContainerStarted","Data":"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99"}
Oct 30 04:17:12.606421 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:12.606353    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3c04cd0d-3f5a-4658-bd0d-8032d4195f55","Type":"ContainerStarted","Data":"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e"}
Oct 30 04:17:12.708933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:12.708891    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.023708835 podStartE2EDuration="6.708877976s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.685169141s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:12.706709651 +0000 UTC m=+5711.713299905" watchObservedRunningTime="2026-10-30 04:17:12.708877976 +0000 UTC m=+5711.715468230"
Oct 30 04:17:15.620871 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:15.620840    4334 generic.go:356] "Generic (PLEG): container finished" podID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerID="50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36" exitCode=0
Oct 30 04:17:15.621208 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:15.620903    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerDied","Data":"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"}
Oct 30 04:17:16.632119 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:16.632083    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerStarted","Data":"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"}
Oct 30 04:17:16.632502 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:16.632154    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerStarted","Data":"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"}
Oct 30 04:17:16.632502 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:16.632181    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerStarted","Data":"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"}
Oct 30 04:17:16.701512 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:16.701442    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=7.415324505 podStartE2EDuration="10.701428752s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.286104247s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:16.689489554 +0000 UTC m=+5715.696079808" watchObservedRunningTime="2026-10-30 04:17:16.701428752 +0000 UTC m=+5715.708019006"
Oct 30 04:17:17.607281 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:17.607248    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:17.608042 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:17.608020    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:20.928432 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:20.928402    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:20.978188 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:20.978152    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:17:25.927975 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:25.927941    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:25.933011 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:25.932991    4334 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:26.682802 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:26.682772    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:34.126063 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.126025    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"]
Oct 30 04:17:34.126604 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.126241    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452" podUID="2fd968fa-c141-49ec-b1e6-1732fa2acca1" containerName="kube-rbac-proxy" containerID="cri-o://b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0" gracePeriod=30
Oct 30 04:17:34.140959 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.140822    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"]
Oct 30 04:17:34.141093 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.141064    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" podUID="2631e41e-25c4-422d-937e-40858114256e" containerName="kube-rbac-proxy" containerID="cri-o://0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95" gracePeriod=30
Oct 30 04:17:34.141229 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.141092    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" podUID="2631e41e-25c4-422d-937e-40858114256e" containerName="prom-label-proxy" containerID="cri-o://06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1" gracePeriod=30
Oct 30 04:17:34.168309 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.168284    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"]
Oct 30 04:17:34.168580 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.168497    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerName="kube-rbac-proxy" containerID="cri-o://224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47" gracePeriod=30
Oct 30 04:17:34.168692 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.168553    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerName="prom-label-proxy" containerID="cri-o://8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b" gracePeriod=30
Oct 30 04:17:34.441424 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.441396    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:34.457747 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.457728    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:34.460861 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.460842    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:34.486874 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.486852    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-client-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-client-ca\") pod \"2631e41e-25c4-422d-937e-40858114256e\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") "
Oct 30 04:17:34.487010 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.486883    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2631e41e-25c4-422d-937e-40858114256e-tls\" (UniqueName: \"kubernetes.io/secret/2631e41e-25c4-422d-937e-40858114256e-tls\") pod \"2631e41e-25c4-422d-937e-40858114256e\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") "
Oct 30 04:17:34.487010 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.486906    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2631e41e-25c4-422d-937e-40858114256e-kube-api-access-b55p6\" (UniqueName: \"kubernetes.io/projected/2631e41e-25c4-422d-937e-40858114256e-kube-api-access-b55p6\") pod \"2631e41e-25c4-422d-937e-40858114256e\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") "
Oct 30 04:17:34.487010 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.486955    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca047f5-45c1-4513-81b8-7e131eed1dae-config\" (UniqueName: \"kubernetes.io/configmap/1ca047f5-45c1-4513-81b8-7e131eed1dae-config\") pod \"1ca047f5-45c1-4513-81b8-7e131eed1dae\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") "
Oct 30 04:17:34.487010 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.486986    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-config\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-config\") pod \"2631e41e-25c4-422d-937e-40858114256e\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") "
Oct 30 04:17:34.487250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487044    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-api-access-m6twl\" (UniqueName: \"kubernetes.io/projected/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-api-access-m6twl\") pod \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") "
Oct 30 04:17:34.487250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487074    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca047f5-45c1-4513-81b8-7e131eed1dae-kube-api-access-8gnm7\" (UniqueName: \"kubernetes.io/projected/1ca047f5-45c1-4513-81b8-7e131eed1dae-kube-api-access-8gnm7\") pod \"1ca047f5-45c1-4513-81b8-7e131eed1dae\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") "
Oct 30 04:17:34.487250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487099    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca047f5-45c1-4513-81b8-7e131eed1dae-tls\" (UniqueName: \"kubernetes.io/secret/1ca047f5-45c1-4513-81b8-7e131eed1dae-tls\") pod \"1ca047f5-45c1-4513-81b8-7e131eed1dae\" (UID: \"1ca047f5-45c1-4513-81b8-7e131eed1dae\") "
Oct 30 04:17:34.487250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487138    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert\") pod \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") "
Oct 30 04:17:34.487250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487167    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-prometheus-tls-ca\") pod \"2631e41e-25c4-422d-937e-40858114256e\" (UID: \"2631e41e-25c4-422d-937e-40858114256e\") "
Oct 30 04:17:34.487250 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487212    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-ca\") pod \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") "
Oct 30 04:17:34.487614 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487266    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-rbac-proxy-config\") pod \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") "
Oct 30 04:17:34.487614 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487306    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-tls\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-tls\") pod \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\" (UID: \"2fd968fa-c141-49ec-b1e6-1732fa2acca1\") "
Oct 30 04:17:34.488653 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487211    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-client-ca" pod "2631e41e-25c4-422d-937e-40858114256e" (UID: "2631e41e-25c4-422d-937e-40858114256e"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:34.488769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.487405    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-config" pod "2631e41e-25c4-422d-937e-40858114256e" (UID: "2631e41e-25c4-422d-937e-40858114256e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:34.488769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.488167    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-ca" pod "2fd968fa-c141-49ec-b1e6-1732fa2acca1" (UID: "2fd968fa-c141-49ec-b1e6-1732fa2acca1"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:34.488769 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.488513    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-prometheus-tls-ca" pod "2631e41e-25c4-422d-937e-40858114256e" (UID: "2631e41e-25c4-422d-937e-40858114256e"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:34.489015 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.488983    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca047f5-45c1-4513-81b8-7e131eed1dae-config" pod "1ca047f5-45c1-4513-81b8-7e131eed1dae" (UID: "1ca047f5-45c1-4513-81b8-7e131eed1dae"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:34.491812 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.491783    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2631e41e-25c4-422d-937e-40858114256e-tls" pod "2631e41e-25c4-422d-937e-40858114256e" (UID: "2631e41e-25c4-422d-937e-40858114256e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:34.491812 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.491795    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert" pod "2fd968fa-c141-49ec-b1e6-1732fa2acca1" (UID: "2fd968fa-c141-49ec-b1e6-1732fa2acca1"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:34.492003 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.491818    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-tls" pod "2fd968fa-c141-49ec-b1e6-1732fa2acca1" (UID: "2fd968fa-c141-49ec-b1e6-1732fa2acca1"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:34.492003 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.491840    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2631e41e-25c4-422d-937e-40858114256e-kube-api-access-b55p6" pod "2631e41e-25c4-422d-937e-40858114256e" (UID: "2631e41e-25c4-422d-937e-40858114256e"). InnerVolumeSpecName "kube-api-access-b55p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:34.492003 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.491874    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-api-access-m6twl" pod "2fd968fa-c141-49ec-b1e6-1732fa2acca1" (UID: "2fd968fa-c141-49ec-b1e6-1732fa2acca1"). InnerVolumeSpecName "kube-api-access-m6twl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:34.492219 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.492202    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca047f5-45c1-4513-81b8-7e131eed1dae-tls" pod "1ca047f5-45c1-4513-81b8-7e131eed1dae" (UID: "1ca047f5-45c1-4513-81b8-7e131eed1dae"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:34.492288 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.492201    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca047f5-45c1-4513-81b8-7e131eed1dae-kube-api-access-8gnm7" pod "1ca047f5-45c1-4513-81b8-7e131eed1dae" (UID: "1ca047f5-45c1-4513-81b8-7e131eed1dae"). InnerVolumeSpecName "kube-api-access-8gnm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:34.493100 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.493041    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-rbac-proxy-config" pod "2fd968fa-c141-49ec-b1e6-1732fa2acca1" (UID: "2fd968fa-c141-49ec-b1e6-1732fa2acca1"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:34.588671 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588627    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6twl\" (UniqueName: \"kubernetes.io/projected/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-api-access-m6twl\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588671 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588671    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gnm7\" (UniqueName: \"kubernetes.io/projected/1ca047f5-45c1-4513-81b8-7e131eed1dae-kube-api-access-8gnm7\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588688    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1ca047f5-45c1-4513-81b8-7e131eed1dae-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588703    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588718    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-prometheus-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588735    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2fd968fa-c141-49ec-b1e6-1732fa2acca1-prometheus-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588749    4334 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-kube-rbac-proxy-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588762    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2fd968fa-c141-49ec-b1e6-1732fa2acca1-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588776    4334 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-client-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588789    4334 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2631e41e-25c4-422d-937e-40858114256e-tls\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588802    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b55p6\" (UniqueName: \"kubernetes.io/projected/2631e41e-25c4-422d-937e-40858114256e-kube-api-access-b55p6\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588815    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ca047f5-45c1-4513-81b8-7e131eed1dae-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.588906 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.588828    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2631e41e-25c4-422d-937e-40858114256e-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:34.697397 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.697350    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:17:34.697812 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.697775    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="prometheus" containerID="cri-o://6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed" gracePeriod=600
Oct 30 04:17:34.698090 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.697825    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="thanos-sidecar" containerID="cri-o://a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982" gracePeriod=600
Oct 30 04:17:34.698090 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.697961    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="config-reloader" containerID="cri-o://ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010" gracePeriod=600
Oct 30 04:17:34.718304 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.718273    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:17:34.719366 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.719298    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="alertmanager" containerID="cri-o://6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e" gracePeriod=120
Oct 30 04:17:34.719479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.719341    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="config-reloader" containerID="cri-o://1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99" gracePeriod=120
Oct 30 04:17:34.719722 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.719694    4334 generic.go:356] "Generic (PLEG): container finished" podID="2fd968fa-c141-49ec-b1e6-1732fa2acca1" containerID="b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0" exitCode=0
Oct 30 04:17:34.719789 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.719756    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"
Oct 30 04:17:34.719789 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.719776    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452" event={"ID":"2fd968fa-c141-49ec-b1e6-1732fa2acca1","Type":"ContainerDied","Data":"b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0"}
Oct 30 04:17:34.719891 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.719817    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452" event={"ID":"2fd968fa-c141-49ec-b1e6-1732fa2acca1","Type":"ContainerDied","Data":"fafbff845809ba6609a015d3ef4228c0529126bce20d36f8ff554a205586665a"}
Oct 30 04:17:34.719891 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.719839    4334 scope.go:122] "RemoveContainer" containerID="b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0"
Oct 30 04:17:34.721843 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.721818    4334 generic.go:356] "Generic (PLEG): container finished" podID="2631e41e-25c4-422d-937e-40858114256e" containerID="06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1" exitCode=0
Oct 30 04:17:34.721843 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.721843    4334 generic.go:356] "Generic (PLEG): container finished" podID="2631e41e-25c4-422d-937e-40858114256e" containerID="0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95" exitCode=0
Oct 30 04:17:34.722075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.721868    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"
Oct 30 04:17:34.722075 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.722024    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" event={"ID":"2631e41e-25c4-422d-937e-40858114256e","Type":"ContainerDied","Data":"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1"}
Oct 30 04:17:34.722196 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.722077    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" event={"ID":"2631e41e-25c4-422d-937e-40858114256e","Type":"ContainerDied","Data":"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95"}
Oct 30 04:17:34.722196 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.722109    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh" event={"ID":"2631e41e-25c4-422d-937e-40858114256e","Type":"ContainerDied","Data":"780829245dd85ca9692bf871ce24e3c6ba0ce16be888739b631020c157472cfc"}
Oct 30 04:17:34.723895 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.723876    4334 generic.go:356] "Generic (PLEG): container finished" podID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerID="8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b" exitCode=0
Oct 30 04:17:34.723895 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.723893    4334 generic.go:356] "Generic (PLEG): container finished" podID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerID="224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47" exitCode=0
Oct 30 04:17:34.724048 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.723920    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" event={"ID":"1ca047f5-45c1-4513-81b8-7e131eed1dae","Type":"ContainerDied","Data":"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b"}
Oct 30 04:17:34.724048 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.723954    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" event={"ID":"1ca047f5-45c1-4513-81b8-7e131eed1dae","Type":"ContainerDied","Data":"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47"}
Oct 30 04:17:34.724048 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.723973    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz" event={"ID":"1ca047f5-45c1-4513-81b8-7e131eed1dae","Type":"ContainerDied","Data":"bcf64f443713901a218f77147e6d2ac3448383459a6e3f6edb26513406193040"}
Oct 30 04:17:34.724048 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.724023    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"
Oct 30 04:17:34.783685 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.783658    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"]
Oct 30 04:17:34.788386 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.788364    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-4m452"]
Oct 30 04:17:34.809426 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.809406    4334 scope.go:122] "RemoveContainer" containerID="b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0"
Oct 30 04:17:34.809777 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:34.809758    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0\": container with ID starting with b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0 not found: ID does not exist" containerID="b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0"
Oct 30 04:17:34.809865 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.809790    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0"} err="failed to get container status \"b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0\": rpc error: code = NotFound desc = could not find container \"b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0\": container with ID starting with b5729f6b7faf3443b38fa87b54734996b5a06317ed7813acc4c2b8f6dad9e9f0 not found: ID does not exist"
Oct 30 04:17:34.809865 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.809813    4334 scope.go:122] "RemoveContainer" containerID="06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1"
Oct 30 04:17:34.819784 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.819761    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"]
Oct 30 04:17:34.829221 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.829198    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-2zlbh"]
Oct 30 04:17:34.833226 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.833210    4334 scope.go:122] "RemoveContainer" containerID="0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95"
Oct 30 04:17:34.840981 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.840963    4334 scope.go:122] "RemoveContainer" containerID="06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1"
Oct 30 04:17:34.841245 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:34.841225    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1\": container with ID starting with 06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1 not found: ID does not exist" containerID="06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1"
Oct 30 04:17:34.841302 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.841257    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1"} err="failed to get container status \"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1\": rpc error: code = NotFound desc = could not find container \"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1\": container with ID starting with 06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1 not found: ID does not exist"
Oct 30 04:17:34.841302 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.841274    4334 scope.go:122] "RemoveContainer" containerID="0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95"
Oct 30 04:17:34.841555 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:34.841537    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95\": container with ID starting with 0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95 not found: ID does not exist" containerID="0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95"
Oct 30 04:17:34.841597 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.841560    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95"} err="failed to get container status \"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95\": rpc error: code = NotFound desc = could not find container \"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95\": container with ID starting with 0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95 not found: ID does not exist"
Oct 30 04:17:34.841597 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.841576    4334 scope.go:122] "RemoveContainer" containerID="06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1"
Oct 30 04:17:34.841799 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.841782    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1"} err="failed to get container status \"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1\": rpc error: code = NotFound desc = could not find container \"06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1\": container with ID starting with 06e9dbb556506fde0aeb9461cd42341e095f8c069081ee34885e2913aa1057c1 not found: ID does not exist"
Oct 30 04:17:34.841849 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.841799    4334 scope.go:122] "RemoveContainer" containerID="0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95"
Oct 30 04:17:34.842013 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.841993    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95"} err="failed to get container status \"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95\": rpc error: code = NotFound desc = could not find container \"0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95\": container with ID starting with 0d44498f47fde16931c3c4e2444018d7fe500b1623fe9b5eee70ae403796bd95 not found: ID does not exist"
Oct 30 04:17:34.842055 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.842014    4334 scope.go:122] "RemoveContainer" containerID="8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b"
Oct 30 04:17:34.849401 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.849384    4334 scope.go:122] "RemoveContainer" containerID="224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47"
Oct 30 04:17:34.856351 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.856337    4334 scope.go:122] "RemoveContainer" containerID="8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b"
Oct 30 04:17:34.856623 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:34.856603    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b\": container with ID starting with 8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b not found: ID does not exist" containerID="8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b"
Oct 30 04:17:34.856714 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.856636    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b"} err="failed to get container status \"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b\": rpc error: code = NotFound desc = could not find container \"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b\": container with ID starting with 8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b not found: ID does not exist"
Oct 30 04:17:34.856714 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.856658    4334 scope.go:122] "RemoveContainer" containerID="224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47"
Oct 30 04:17:34.856901 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:34.856882    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47\": container with ID starting with 224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47 not found: ID does not exist" containerID="224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47"
Oct 30 04:17:34.856941 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.856906    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47"} err="failed to get container status \"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47\": rpc error: code = NotFound desc = could not find container \"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47\": container with ID starting with 224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47 not found: ID does not exist"
Oct 30 04:17:34.856941 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.856921    4334 scope.go:122] "RemoveContainer" containerID="8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b"
Oct 30 04:17:34.857129 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.857111    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b"} err="failed to get container status \"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b\": rpc error: code = NotFound desc = could not find container \"8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b\": container with ID starting with 8e54e3aed26e94f5887cc730bbafcd642dde3befabbb0917903475abdb7f7c4b not found: ID does not exist"
Oct 30 04:17:34.857201 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.857130    4334 scope.go:122] "RemoveContainer" containerID="224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47"
Oct 30 04:17:34.857390 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.857371    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47"} err="failed to get container status \"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47\": rpc error: code = NotFound desc = could not find container \"224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47\": container with ID starting with 224444876fd4e406cb7c7057f726553686f24c555e59b25f11baa793975fce47 not found: ID does not exist"
Oct 30 04:17:34.863797 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.863777    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"]
Oct 30 04:17:34.868080 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:34.868057    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-cclpz"]
Oct 30 04:17:35.159016 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.158984    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" path="/var/lib/kubelet/pods/1ca047f5-45c1-4513-81b8-7e131eed1dae/volumes"
Oct 30 04:17:35.159406 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.159392    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2631e41e-25c4-422d-937e-40858114256e" path="/var/lib/kubelet/pods/2631e41e-25c4-422d-937e-40858114256e/volumes"
Oct 30 04:17:35.159800 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.159787    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd968fa-c141-49ec-b1e6-1732fa2acca1" path="/var/lib/kubelet/pods/2fd968fa-c141-49ec-b1e6-1732fa2acca1/volumes"
Oct 30 04:17:35.343430 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.343393    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:35.343666 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.343643    4334 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" containerName="otc-container" containerID="cri-o://16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1" gracePeriod=30
Oct 30 04:17:35.557379 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.557126    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:35.598267 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.598235    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:35.611792 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.611762    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.611927 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.611814    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.611927 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.611858    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-secret-prometheus-web-tls-ca\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.611927 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.611890    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-tls-assets\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-tls-assets\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.611934    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-thanos-prometheus-http-client-file\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.611968    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.611996    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.612027    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.612055    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config-out\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612355 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.612129    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612355 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.612221    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612355 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.612251    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-kube-api-access-c2ck9\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-kube-api-access-c2ck9\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.612355 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.612289    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\" (UID: \"e1ede8f0-7894-4b2c-9436-b0db653b8fcf\") "
Oct 30 04:17:35.619475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.613866    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-0" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.619475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.615467    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-1" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.619475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.616709    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.619475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.616973    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-2" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.619475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.618085    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config-out" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:35.619475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.619011    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-thanos-prometheus-http-client-file" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.619475 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.619091    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-kube-api-access-c2ck9" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "kube-api-access-c2ck9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:35.620603 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.620576    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.621922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.621885    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-secret-prometheus-web-tls-ca" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.622006 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.621960    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.623028 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.623009    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:17:35.625111 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.625089    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-tls-assets" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:35.634293 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.634272    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config" pod "e1ede8f0-7894-4b2c-9436-b0db653b8fcf" (UID: "e1ede8f0-7894-4b2c-9436-b0db653b8fcf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.673717 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.673658    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:35.713664 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.713640    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-tls-assets\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-tls-assets\") pod \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") "
Oct 30 04:17:35.713818 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.713682    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-data-science-collector-ta-client-cert\") pod \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") "
Oct 30 04:17:35.713818 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.713727    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-out\") pod \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") "
Oct 30 04:17:35.713818 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.713762    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-web-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-web-config\") pod \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") "
Oct 30 04:17:35.713818 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.713793    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-alertmanager-data-science-monitoringstack-db\") pod \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714115    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-alertmanager-data-science-monitoringstack-db" pod "3c04cd0d-3f5a-4658-bd0d-8032d4195f55" (UID: "3c04cd0d-3f5a-4658-bd0d-8032d4195f55"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714205    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otc-internal\") pod \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714268    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-monitoring-tls-certs\") pod \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714313    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-kube-api-access-v799g\" (UniqueName: \"kubernetes.io/projected/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-kube-api-access-v799g\") pod \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714346    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-volume\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-volume\") pod \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714373    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-kube-api-access-8mmgv\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-kube-api-access-8mmgv\") pod \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714410    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otlp-tls-certs\") pod \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714489    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-cluster-tls-config\") pod \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\" (UID: \"3c04cd0d-3f5a-4658-bd0d-8032d4195f55\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714518    4334 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-tls-certs\") pod \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\" (UID: \"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3\") "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714572    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otc-internal" pod "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" (UID: "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714902    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714923    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714957    4334 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714972    4334 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otc-internal\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.714988    4334 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715034    4334 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" "
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715050    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2ck9\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-kube-api-access-c2ck9\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715065    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715081    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715115    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715130    4334 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-secret-prometheus-web-tls-ca\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715146    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715176    4334 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715194    4334 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.715258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.715211    4334 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e1ede8f0-7894-4b2c-9436-b0db653b8fcf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.718858 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.716392    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-data-science-collector-ta-client-cert" pod "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" (UID: "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.718858 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.716474    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-tls-assets" pod "3c04cd0d-3f5a-4658-bd0d-8032d4195f55" (UID: "3c04cd0d-3f5a-4658-bd0d-8032d4195f55"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:35.718858 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.717680    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-out" pod "3c04cd0d-3f5a-4658-bd0d-8032d4195f55" (UID: "3c04cd0d-3f5a-4658-bd0d-8032d4195f55"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:35.718858 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.717844    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-volume" pod "3c04cd0d-3f5a-4658-bd0d-8032d4195f55" (UID: "3c04cd0d-3f5a-4658-bd0d-8032d4195f55"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.720525 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.719198    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-kube-api-access-v799g" pod "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" (UID: "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3"). InnerVolumeSpecName "kube-api-access-v799g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:35.720706 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.720599    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-monitoring-tls-certs" pod "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" (UID: "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.720706 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.720672    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otlp-tls-certs" pod "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" (UID: "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.721188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.721164    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-kube-api-access-8mmgv" pod "3c04cd0d-3f5a-4658-bd0d-8032d4195f55" (UID: "3c04cd0d-3f5a-4658-bd0d-8032d4195f55"). InnerVolumeSpecName "kube-api-access-8mmgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:35.723212 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.723183    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-tls-certs" pod "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" (UID: "8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.725761 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.725742    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-cluster-tls-config" pod "3c04cd0d-3f5a-4658-bd0d-8032d4195f55" (UID: "3c04cd0d-3f5a-4658-bd0d-8032d4195f55"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.732956 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.732932    4334 generic.go:356] "Generic (PLEG): container finished" podID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerID="a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982" exitCode=0
Oct 30 04:17:35.732956 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.732955    4334 generic.go:356] "Generic (PLEG): container finished" podID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerID="ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010" exitCode=0
Oct 30 04:17:35.733122 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.732965    4334 generic.go:356] "Generic (PLEG): container finished" podID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerID="6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed" exitCode=0
Oct 30 04:17:35.733122 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.733004    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:17:35.733122 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.733010    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerDied","Data":"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"}
Oct 30 04:17:35.733122 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.733079    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerDied","Data":"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"}
Oct 30 04:17:35.733344 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.733132    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerDied","Data":"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"}
Oct 30 04:17:35.733344 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.733162    4334 scope.go:122] "RemoveContainer" containerID="a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"
Oct 30 04:17:35.733344 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.733170    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e1ede8f0-7894-4b2c-9436-b0db653b8fcf","Type":"ContainerDied","Data":"cfdbd39a0b87149ca25dc1d80b92b2b92c95a52ad3d9b3170abbf099c6822a38"}
Oct 30 04:17:35.734019 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.733999    4334 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-72ca8347-922a-464b-8594-dfac31ff3c9d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2") on node "ip-10-0-133-197.ec2.internal"
Oct 30 04:17:35.734305 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.734284    4334 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-web-config" pod "3c04cd0d-3f5a-4658-bd0d-8032d4195f55" (UID: "3c04cd0d-3f5a-4658-bd0d-8032d4195f55"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.736030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.736008    4334 generic.go:356] "Generic (PLEG): container finished" podID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerID="1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99" exitCode=0
Oct 30 04:17:35.736030 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.736028    4334 generic.go:356] "Generic (PLEG): container finished" podID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerID="6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e" exitCode=0
Oct 30 04:17:35.736188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.736082    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:17:35.736188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.736095    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3c04cd0d-3f5a-4658-bd0d-8032d4195f55","Type":"ContainerDied","Data":"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99"}
Oct 30 04:17:35.736188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.736149    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3c04cd0d-3f5a-4658-bd0d-8032d4195f55","Type":"ContainerDied","Data":"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e"}
Oct 30 04:17:35.736188 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.736181    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3c04cd0d-3f5a-4658-bd0d-8032d4195f55","Type":"ContainerDied","Data":"2eee8a24da091de3c1c8b89d3fcdaad2de98662746e830251e18b955fd73690d"}
Oct 30 04:17:35.741159 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.741137    4334 generic.go:356] "Generic (PLEG): container finished" podID="8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" containerID="16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1" exitCode=0
Oct 30 04:17:35.741249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.741164    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3","Type":"ContainerDied","Data":"16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1"}
Oct 30 04:17:35.741249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.741197    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3","Type":"ContainerDied","Data":"f8c1d85a7817c35e704edf877aa0bf4dd126b3f3400b9bd288c335ec177851a5"}
Oct 30 04:17:35.741249 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.741202    4334 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:35.745589 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.745566    4334 scope.go:122] "RemoveContainer" containerID="ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"
Oct 30 04:17:35.754091 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.754076    4334 scope.go:122] "RemoveContainer" containerID="6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"
Oct 30 04:17:35.761729 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.761711    4334 scope.go:122] "RemoveContainer" containerID="50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"
Oct 30 04:17:35.769185 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.769169    4334 scope.go:122] "RemoveContainer" containerID="a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"
Oct 30 04:17:35.769534 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:35.769511    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982\": container with ID starting with a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982 not found: ID does not exist" containerID="a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"
Oct 30 04:17:35.769676 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.769537    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"} err="failed to get container status \"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982\": rpc error: code = NotFound desc = could not find container \"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982\": container with ID starting with a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982 not found: ID does not exist"
Oct 30 04:17:35.769676 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.769554    4334 scope.go:122] "RemoveContainer" containerID="ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"
Oct 30 04:17:35.769806 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:35.769790    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010\": container with ID starting with ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010 not found: ID does not exist" containerID="ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"
Oct 30 04:17:35.769846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.769811    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"} err="failed to get container status \"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010\": rpc error: code = NotFound desc = could not find container \"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010\": container with ID starting with ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010 not found: ID does not exist"
Oct 30 04:17:35.769846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.769826    4334 scope.go:122] "RemoveContainer" containerID="6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"
Oct 30 04:17:35.770047 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:35.770031    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed\": container with ID starting with 6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed not found: ID does not exist" containerID="6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"
Oct 30 04:17:35.770088 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770051    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"} err="failed to get container status \"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed\": rpc error: code = NotFound desc = could not find container \"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed\": container with ID starting with 6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed not found: ID does not exist"
Oct 30 04:17:35.770088 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770065    4334 scope.go:122] "RemoveContainer" containerID="50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"
Oct 30 04:17:35.770304 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:35.770289    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36\": container with ID starting with 50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36 not found: ID does not exist" containerID="50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"
Oct 30 04:17:35.770350 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770307    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"} err="failed to get container status \"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36\": rpc error: code = NotFound desc = could not find container \"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36\": container with ID starting with 50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36 not found: ID does not exist"
Oct 30 04:17:35.770350 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770319    4334 scope.go:122] "RemoveContainer" containerID="a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"
Oct 30 04:17:35.770533 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770518    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"} err="failed to get container status \"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982\": rpc error: code = NotFound desc = could not find container \"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982\": container with ID starting with a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982 not found: ID does not exist"
Oct 30 04:17:35.770587 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770532    4334 scope.go:122] "RemoveContainer" containerID="ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"
Oct 30 04:17:35.770740 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770723    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"} err="failed to get container status \"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010\": rpc error: code = NotFound desc = could not find container \"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010\": container with ID starting with ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010 not found: ID does not exist"
Oct 30 04:17:35.770740 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770739    4334 scope.go:122] "RemoveContainer" containerID="6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"
Oct 30 04:17:35.770953 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770930    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"} err="failed to get container status \"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed\": rpc error: code = NotFound desc = could not find container \"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed\": container with ID starting with 6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed not found: ID does not exist"
Oct 30 04:17:35.770953 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.770952    4334 scope.go:122] "RemoveContainer" containerID="50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"
Oct 30 04:17:35.771155 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.771138    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"} err="failed to get container status \"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36\": rpc error: code = NotFound desc = could not find container \"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36\": container with ID starting with 50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36 not found: ID does not exist"
Oct 30 04:17:35.771155 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.771153    4334 scope.go:122] "RemoveContainer" containerID="a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"
Oct 30 04:17:35.771363 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.771346    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982"} err="failed to get container status \"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982\": rpc error: code = NotFound desc = could not find container \"a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982\": container with ID starting with a0771180cdad008b75fff347e45b1617c1573443ed7579d59e89774aa7444982 not found: ID does not exist"
Oct 30 04:17:35.771422 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.771363    4334 scope.go:122] "RemoveContainer" containerID="ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"
Oct 30 04:17:35.771634 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.771610    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010"} err="failed to get container status \"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010\": rpc error: code = NotFound desc = could not find container \"ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010\": container with ID starting with ff2780819f68b6a952e16108212896c665ca98069ff261aa5a83f3b16d74a010 not found: ID does not exist"
Oct 30 04:17:35.771695 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.771635    4334 scope.go:122] "RemoveContainer" containerID="6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"
Oct 30 04:17:35.771839 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.771823    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed"} err="failed to get container status \"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed\": rpc error: code = NotFound desc = could not find container \"6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed\": container with ID starting with 6db178e85263a20edb60570ac618e07f398cc9c2feb7981d61b89084513cc5ed not found: ID does not exist"
Oct 30 04:17:35.771901 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.771838    4334 scope.go:122] "RemoveContainer" containerID="50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"
Oct 30 04:17:35.772065 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.772048    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36"} err="failed to get container status \"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36\": rpc error: code = NotFound desc = could not find container \"50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36\": container with ID starting with 50bd4b0bb62af9aa880d50db32becc6b4b818142f33e11c767a58db5e372cc36 not found: ID does not exist"
Oct 30 04:17:35.772112 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.772065    4334 scope.go:122] "RemoveContainer" containerID="1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99"
Oct 30 04:17:35.779092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.779075    4334 scope.go:122] "RemoveContainer" containerID="6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e"
Oct 30 04:17:35.786140 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.786122    4334 scope.go:122] "RemoveContainer" containerID="b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8"
Oct 30 04:17:35.793417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.793403    4334 scope.go:122] "RemoveContainer" containerID="1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99"
Oct 30 04:17:35.793695 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:35.793678    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99\": container with ID starting with 1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99 not found: ID does not exist" containerID="1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99"
Oct 30 04:17:35.793740 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.793700    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99"} err="failed to get container status \"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99\": rpc error: code = NotFound desc = could not find container \"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99\": container with ID starting with 1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99 not found: ID does not exist"
Oct 30 04:17:35.793740 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.793716    4334 scope.go:122] "RemoveContainer" containerID="6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e"
Oct 30 04:17:35.793955 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:35.793938    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e\": container with ID starting with 6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e not found: ID does not exist" containerID="6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e"
Oct 30 04:17:35.793997 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.793959    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e"} err="failed to get container status \"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e\": rpc error: code = NotFound desc = could not find container \"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e\": container with ID starting with 6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e not found: ID does not exist"
Oct 30 04:17:35.793997 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.793973    4334 scope.go:122] "RemoveContainer" containerID="b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8"
Oct 30 04:17:35.794199 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:35.794182    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8\": container with ID starting with b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8 not found: ID does not exist" containerID="b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8"
Oct 30 04:17:35.794235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.794203    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8"} err="failed to get container status \"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8\": rpc error: code = NotFound desc = could not find container \"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8\": container with ID starting with b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8 not found: ID does not exist"
Oct 30 04:17:35.794235 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.794214    4334 scope.go:122] "RemoveContainer" containerID="1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99"
Oct 30 04:17:35.794417 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.794400    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99"} err="failed to get container status \"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99\": rpc error: code = NotFound desc = could not find container \"1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99\": container with ID starting with 1f7418b260a3d6193068ad53a13db86a6abf479c3c5c888ca843f8eada0feb99 not found: ID does not exist"
Oct 30 04:17:35.794479 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.794417    4334 scope.go:122] "RemoveContainer" containerID="6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e"
Oct 30 04:17:35.794749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.794732    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e"} err="failed to get container status \"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e\": rpc error: code = NotFound desc = could not find container \"6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e\": container with ID starting with 6cd43f60401ba6a43347f90df01180125bcb2068b6ab6610f6a2279b5d9dcf3e not found: ID does not exist"
Oct 30 04:17:35.794749 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.794747    4334 scope.go:122] "RemoveContainer" containerID="b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8"
Oct 30 04:17:35.794969 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.794952    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8"} err="failed to get container status \"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8\": rpc error: code = NotFound desc = could not find container \"b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8\": container with ID starting with b5b3571af9d16b7ffbb944fed3cf6e72d6a46e93f0d5866ca7b2d428f7df52c8 not found: ID does not exist"
Oct 30 04:17:35.795014 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.794968    4334 scope.go:122] "RemoveContainer" containerID="16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1"
Oct 30 04:17:35.801695 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.801680    4334 scope.go:122] "RemoveContainer" containerID="16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1"
Oct 30 04:17:35.801944 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:35.801924    4334 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1\": container with ID starting with 16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1 not found: ID does not exist" containerID="16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1"
Oct 30 04:17:35.801997 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.801951    4334 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1"} err="failed to get container status \"16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1\": rpc error: code = NotFound desc = could not find container \"16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1\": container with ID starting with 16fe553d8ee8a84cba0870e9ff0f609d88f757c4147e1e5fd45d2953306190c1 not found: ID does not exist"
Oct 30 04:17:35.815604 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815588    4334 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-tls-assets\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815606    4334 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-data-science-collector-ta-client-cert\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815615    4334 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-out\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815624    4334 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-web-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815633    4334 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-monitoring-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815641    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v799g\" (UniqueName: \"kubernetes.io/projected/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-kube-api-access-v799g\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815650    4334 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-config-volume\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815658    4334 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmgv\" (UniqueName: \"kubernetes.io/projected/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-kube-api-access-8mmgv\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815669    4334 reconciler_common.go:299] "Volume detached for volume \"pvc-72ca8347-922a-464b-8594-dfac31ff3c9d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b60791c665fd0ae2\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"csi-eaedccda45e81dc7cfa587f5d76be90aaea3b3423ef6a631dccfb966a06a97d8\""
Oct 30 04:17:35.815680 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815678    4334 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-otlp-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815968 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815686    4334 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c04cd0d-3f5a-4658-bd0d-8032d4195f55-cluster-tls-config\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.815968 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.815694    4334 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3-tls-certs\") on node \"ip-10-0-133-197.ec2.internal\" DevicePath \"\""
Oct 30 04:17:35.822360 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.822333    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:17:35.845498 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.845479    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:17:35.907260 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.907232    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:35.935092 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.935029    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:35.969571 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:35.969543    4334 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:17:36.001976 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:36.001939    4334 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:17:37.158008 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:37.157977    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" path="/var/lib/kubelet/pods/3c04cd0d-3f5a-4658-bd0d-8032d4195f55/volumes"
Oct 30 04:17:37.158423 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:37.158410    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" path="/var/lib/kubelet/pods/8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3/volumes"
Oct 30 04:17:37.158831 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:17:37.158819    4334 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" path="/var/lib/kubelet/pods/e1ede8f0-7894-4b2c-9436-b0db653b8fcf/volumes"
Oct 30 04:17:50.978460 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:17:50.978408    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:18:14.361297 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:14.361267    4334 ???:1] "http: TLS handshake error from 10.0.128.241:49358: EOF"
Oct 30 04:18:14.380741 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:14.380712    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6xh6w_033b7d72-7e35-47ee-9cf9-d9993d21f4be/global-pull-secret-syncer/0.log"
Oct 30 04:18:15.536862 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:15.536837    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mkxt8_172aafae-6ad9-420e-a213-a8bc8f2439fe/konnectivity-agent/0.log"
Oct 30 04:18:15.958514 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:15.958487    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-197.ec2.internal_adf118574a1cbe5f944e98682e2ff56b/haproxy/0.log"
Oct 30 04:18:20.977749 ip-10-0-133-197 kubenswrapper[4334]: E1030 04:18:20.977712    4334 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bvj8c" containerName="extractor"
Oct 30 04:18:22.364443 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:22.364404    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4wz5_ce6fe84e-ade4-48a8-8737-5767fe363a66/node-exporter/0.log"
Oct 30 04:18:22.484488 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:22.484463    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4wz5_ce6fe84e-ade4-48a8-8737-5767fe363a66/kube-rbac-proxy/0.log"
Oct 30 04:18:22.606276 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:22.606252    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4wz5_ce6fe84e-ade4-48a8-8737-5767fe363a66/init-textfile/0.log"
Oct 30 04:18:23.428333 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428300    4334 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"]
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428625    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2fd968fa-c141-49ec-b1e6-1732fa2acca1" containerName="kube-rbac-proxy"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428644    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd968fa-c141-49ec-b1e6-1732fa2acca1" containerName="kube-rbac-proxy"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428664    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerName="kube-rbac-proxy"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428669    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerName="kube-rbac-proxy"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428677    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="alertmanager"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428682    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="alertmanager"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428687    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="init-config-reloader"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428692    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="init-config-reloader"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428697    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="thanos-sidecar"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428702    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="thanos-sidecar"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428711    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2631e41e-25c4-422d-937e-40858114256e" containerName="kube-rbac-proxy"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428716    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2631e41e-25c4-422d-937e-40858114256e" containerName="kube-rbac-proxy"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428722    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="config-reloader"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428727    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="config-reloader"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428734    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="prometheus"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428738    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="prometheus"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428750    4334 cpu_manager.go:414] "containerMap: removing container" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerName="prom-label-proxy"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428757    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerName="prom-label-proxy"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428764    4334 cpu_manager.go:414] "containerMap: removing container" podUID="8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" containerName="otc-container"
Oct 30 04:18:23.428758 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428769    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" containerName="otc-container"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428780    4334 cpu_manager.go:414] "containerMap: removing container" podUID="2631e41e-25c4-422d-937e-40858114256e" containerName="prom-label-proxy"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428784    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2631e41e-25c4-422d-937e-40858114256e" containerName="prom-label-proxy"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428791    4334 cpu_manager.go:414] "containerMap: removing container" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="config-reloader"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428795    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="config-reloader"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428804    4334 cpu_manager.go:414] "containerMap: removing container" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="init-config-reloader"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428810    4334 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="init-config-reloader"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428874    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerName="kube-rbac-proxy"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428885    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="alertmanager"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428895    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="thanos-sidecar"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428905    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2631e41e-25c4-422d-937e-40858114256e" containerName="kube-rbac-proxy"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428913    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c04cd0d-3f5a-4658-bd0d-8032d4195f55" containerName="config-reloader"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428918    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="prometheus"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428924    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca047f5-45c1-4513-81b8-7e131eed1dae" containerName="prom-label-proxy"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428930    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1cf36a-6e4d-4aa4-ba73-5609b4db5db3" containerName="otc-container"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428937    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2631e41e-25c4-422d-937e-40858114256e" containerName="prom-label-proxy"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428942    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ede8f0-7894-4b2c-9436-b0db653b8fcf" containerName="config-reloader"
Oct 30 04:18:23.429641 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.428948    4334 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd968fa-c141-49ec-b1e6-1732fa2acca1" containerName="kube-rbac-proxy"
Oct 30 04:18:23.431837 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.431821    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.446632 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.446608    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"]
Oct 30 04:18:23.456455 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.456422    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6xn7p\"/\"kube-root-ca.crt\""
Oct 30 04:18:23.456455 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.456433    4334 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6xn7p\"/\"openshift-service-ca.crt\""
Oct 30 04:18:23.456636 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.456535    4334 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6xn7p\"/\"default-dockercfg-4j88g\""
Oct 30 04:18:23.562968 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.562938    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-lib-modules\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.563111 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.562986    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-sys\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.563111 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.563079    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnx6s\" (UniqueName: \"kubernetes.io/projected/127ecd3e-57ba-4322-bcf7-0f1d265dff78-kube-api-access-tnx6s\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.563111 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.563108    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-podres\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.563231 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.563126    4334 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-proc\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663663 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663634    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnx6s\" (UniqueName: \"kubernetes.io/projected/127ecd3e-57ba-4322-bcf7-0f1d265dff78-kube-api-access-tnx6s\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663669    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-podres\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663689    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-proc\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663717    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-lib-modules\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663750    4334 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-sys\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663772    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-proc\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663777    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-podres\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663837    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-lib-modules\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.663846 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.663839    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/127ecd3e-57ba-4322-bcf7-0f1d265dff78-sys\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.681936 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.681882    4334 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnx6s\" (UniqueName: \"kubernetes.io/projected/127ecd3e-57ba-4322-bcf7-0f1d265dff78-kube-api-access-tnx6s\") pod \"perf-node-gather-daemonset-zjrfc\" (UID: \"127ecd3e-57ba-4322-bcf7-0f1d265dff78\") " pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:23.743947 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:23.743912    4334 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:24.038175 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:24.038103    4334 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"]
Oct 30 04:18:24.042985 ip-10-0-133-197 kubenswrapper[4334]: W1030 04:18:24.042959    4334 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod127ecd3e_57ba_4322_bcf7_0f1d265dff78.slice/crio-57f69be7dfefd4f0c871204aa0542863d2a111ff7f0b3259884df1b1d4b9741c WatchSource:0}: Error finding container 57f69be7dfefd4f0c871204aa0542863d2a111ff7f0b3259884df1b1d4b9741c: Status 404 returned error can't find the container with id 57f69be7dfefd4f0c871204aa0542863d2a111ff7f0b3259884df1b1d4b9741c
Oct 30 04:18:24.886263 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:24.886231    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-hds4n_becf929b-932d-4100-bc01-0adec847862b/prometheus-operator-admission-webhook/0.log"
Oct 30 04:18:24.963855 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:24.963826    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc" event={"ID":"127ecd3e-57ba-4322-bcf7-0f1d265dff78","Type":"ContainerStarted","Data":"f31fba82004f3e47afcffe4461e231fdb4d59bb7dc458275896eeb8daf1f8231"}
Oct 30 04:18:24.964007 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:24.963873    4334 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc" event={"ID":"127ecd3e-57ba-4322-bcf7-0f1d265dff78","Type":"ContainerStarted","Data":"57f69be7dfefd4f0c871204aa0542863d2a111ff7f0b3259884df1b1d4b9741c"}
Oct 30 04:18:24.964007 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:24.963925    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:24.999528 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:24.999489    4334 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc" podStartSLOduration=1.9994779280000001 podStartE2EDuration="1.999477928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:24.998339773 +0000 UTC m=+5784.004930027" watchObservedRunningTime="2026-10-30 04:18:24.999477928 +0000 UTC m=+5784.006068185"
Oct 30 04:18:25.015096 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:25.015070    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bd95bbcdd-n9zhl_473e27f6-e488-472b-a0b1-4ede62f08ba1/telemeter-client/0.log"
Oct 30 04:18:25.138258 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:25.138200    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bd95bbcdd-n9zhl_473e27f6-e488-472b-a0b1-4ede62f08ba1/reload/0.log"
Oct 30 04:18:25.259106 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:25.259085    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bd95bbcdd-n9zhl_473e27f6-e488-472b-a0b1-4ede62f08ba1/kube-rbac-proxy/0.log"
Oct 30 04:18:29.051954 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:29.051927    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-545cf8b9c5-b5tb6_434cbe69-1f14-42f2-bd9c-446dd3a786fb/console/0.log"
Oct 30 04:18:29.191723 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:29.191698    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-j8m67_2a14bb01-c443-4ddb-910f-2ef20f279ba5/download-server/0.log"
Oct 30 04:18:30.979222 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:30.979196    4334 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6xn7p/perf-node-gather-daemonset-zjrfc"
Oct 30 04:18:32.458312 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:32.458280    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8k7nc_1436ee1e-5bbf-488b-812b-92c7bc0906a2/dns/0.log"
Oct 30 04:18:32.580806 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:32.580777    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8k7nc_1436ee1e-5bbf-488b-812b-92c7bc0906a2/kube-rbac-proxy/0.log"
Oct 30 04:18:33.047655 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:33.047620    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k5blc_818913a6-9255-4e66-b325-5ced6778ad64/dns-node-resolver/0.log"
Oct 30 04:18:34.212933 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:34.212908    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d86776dcf-bqt2x_f9869074-3251-4428-852a-30d9058ec7ab/registry/0.log"
Oct 30 04:18:34.486772 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:34.486697    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qb86g_1628ebb0-5634-4a42-bca2-b09ecdbfd6be/node-ca/0.log"
Oct 30 04:18:37.205080 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:37.205053    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5tmj8_c080eab2-cfb6-4495-ba04-475ad7344a43/serve-healthcheck-canary/0.log"
Oct 30 04:18:38.405802 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:38.405771    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvj8c_180f424e-5860-452d-98b6-c0d4585dd13c/kube-rbac-proxy/0.log"
Oct 30 04:18:38.525922 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:38.525899    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvj8c_180f424e-5860-452d-98b6-c0d4585dd13c/exporter/0.log"
Oct 30 04:18:38.646376 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:38.646348    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bvj8c_180f424e-5860-452d-98b6-c0d4585dd13c/extractor/0.log"
Oct 30 04:18:39.491668 ip-10-0-133-197 kubenswrapper[4334]: I1030 04:18:39.491638    4334 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-zvcd8-dqps8_4f962975-75bf-409b-b0e5-a340780bffa5/insights-gathering/0.log"